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ABSTRACT 
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develop a research, development, and applications program for the use 
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training, performance measurement, and job performance aiding. The 
paper is organized in five sections that (1) introduce the reader to 
AI and those subfields of AI that are most relevant to this program, 
(2) report on relevant ongoing research and development sponsored by 
the Department of Defense, (3) establish the challenges facing the 
Air Force in three target areas and explain how each presents 
opportunities for AI applications, (4) draw out important practical 
concerns that must be faced by an AI research and development 
program, and finally, (5) propose a set of recommendations for 
building an Air Force AI applications program in training, 
performance measurement, and job performance aiding. An eight-page 
bibliography and a list of current relevant Department of Defense 
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CHAPTER I. INTRODUCTION 



icon u C,al intelli 8 ence < AI > « one of the emerging technoiogies of the 

1980s. AI is the use of computers to perform tasks typically considered to require 
intelligence. This form of computing concerns the manipulation of symbols which 
encode the knowledge and the competence to perform intelligert acts, rather th&n 
the familiar form of computing that concerns numerical calculation and r^co-d 
keeping. 

Almost all computing done today involves numerical calculation; for 
example, work in the physical sciences and engineering. However, a vast amount 
of human productivity involves symbol manipulation: solving legal problems, 
making medical diagnoses, designing systems, or planning genetic engineering 
experiments. Edward Feigenbaum has said (S*vaine, 1983, p. 12), "Symbolic 
manipulation will become the dominant type o. omputer use simply because 

„. re u? re u $0 many more things to done omparet vith mere calculation." When 
will this happen? Feigenbaum predicts, "if not by 1990, perhaps by the turn of the 
century. Think of the absolutely vast amount of computation that goes on today, 
and then imagine that amount of computing, or more, being devoted to symbol 
manipulation. This is the future of AL 

At this level of generality, tb- requirements of the Air Force and of the 
other Arm-d Services are no different from society at large: much work that 
needs to be done is symbolic in nature. This report focuses on computer 
applications to the inherently symbolic (as contrast to computational) tasks of 
designing, developing and delivering instruction; monitoring and assessing 
personnel performance; and assisting personnel in the performance of their jobs. 
That is, the operations and maintenance of Air Force systems provide both 
challenges and opportunities for the application of AI technology. 

Background of This Report 

This paper is the first step in a 9-month program planning effort, tasked 
to develop a research, development, and applications program for the utilization 
of artificial intelligence technology in training, performance assessment and job 
performance aiding. The objectives of this paper are (a) to introduce the reader 
to A I and those subfields of AI that are most relevant to this research and 
development (R&D) program, (b) to report on relevant ongoing. R<5cD sponsored by 
the Department of Defense (DoD), (c) to establish the challenges facing the Air 
Force in the three target areas and explain how each presents opportunities for AI 
applications, (d) to draw out important practical concerns which must be faced by 
an A I R&D program, and finally, (e) to draw this analysis together by proposing a 
set of recommendations for building an AI applications program in training 
performance assessment and job performance aiding. 
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Capsule History of AI 



The brief history of AI spans about 25 years. Early projections of 
competent AI applications in language translation and chess were 
disappointments* AI proved to be much harder than expected. Understandably, a 
backlash of skepticism wus generated by these early failures. But, AI profited 
from experience. Its re?*arch thrust was redirected. It was recognized that 
computational brute force and generality are not the keys to solving important 
problems. The beginning of Ars success story was the recognition that knowledge 
powers the solution to important problems. 

By the late 1970s, this redirection brought success to AI research 
endeavors. AI programs were developed that performed successfully or 
demonstrated promise in a number of different applications areas; for example, 
medical diagnosis, speech understanding, spectroscopy, and symbolic 
mathematics. By 1980, the potential of AI was recognized, and active AI research 
programs were underway in several » age corporations. In 1981, Japan announced 
a major commitment to AI applications with its fifth generation computer 
project. The first AI applications business ventures were launched. By 1982, AI 
could be considered to be entering a period of industrialization and public 
awareness. Numerous AI business ventures had been launched and the promise of 
AI had been touted in ai least six major broad circuiation periodicals. 

Thus, AI's time has come. An examination of potential AI applications in 
the DoD in general, and in training, performance assessment, and job performance 
aiding in particular, is timely and justifiable. 

Artificial Intelligence 

Definition 

AI is the study and appiication of what is known about intelligence to the 
development of computer systems that model intelligent behavior. Intelligent 
behavior involves tasks for which algorithmic solutions do not exist. Such tasks 
often involve complexity (designing a bridge), uncertainty (deciding whether to 
buy or sell on today's stock market) or ambiguity (filling in the details whiie 
reading a novel). Usually, a controlled search, rather than a random or exhaustive 
search, for a solution is necessary. Great amounts of knowledge are typically 
required for successful performance. AI is concerned with several types of 
knowledge, principally procedural knowledge (how-to-do-it), declarative 
knowledge (knowledge of function or structure), and control knowledge (general 
problem solving strategies). 

Research in AI is being conducted in two disciplines with complementary 
objectives: computer science and cognitive science. As a branch of computer 
science, the study of AI seeks to develop new tools and new techniques that will 
support the use of computers to do tasks that require intelligence. As a major 
branch of the emerging discipline of cognitive science, the study of AI serves to 
provide controlled environments for experimenting with methods and tools used in 
studying human intelligent behavior. 
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The major source of knowledfc,<! (about intelligence) is the 
experimental study of human behavior to discover 
mechanisms, followed by construction of simulation systems 
(e.g., computer programs) to verify whether the discovered 
mechanisms do indeed have the properties inferred. (Newell. 
1973, p< 1^.) ' 

As humans, we are our own best exemplars of intelligent behavior. By 
following our example, we are learning how to teach machines to help us do our 
work* 



AI Elements 

Representing knowledge and techniques for processing knowledge are the 
two basic elements of AI. 

Depicting, symbolizing, structuring and communicating knowledge have 
clearly become a focus in the short history of AI: 

The fundamental problem of understanding intelligence is not 
the identification of a few powerful techniques, but rather 
the question of how to represent large amounts of knowledge 
in a fashion that permits their effective use and interaction. 
(Goldstein & Papert, 1977, p. 85) 

The second element, processes that operate on representations, is 
predicated on the principle that intelligent behavior is objective-directed or goal- 
seeking (e.g., in problem solving ... a solution; in speech understanding. . . 
meaning; m medical consultation ... a correct diapiosuj and prescription). The 
attempt to reach a goal requires first creating, then searching an "area" or space 
within which the goal will lie. Creating the space requires the building of a 
representation of the problem of situation. In chess this representation could be a 
tree of all possible ai moves and countei-moves. After creatine a 
representation, a scare the space is conducted. Controlling search is a major 
AI issue. ' 



Traditional AI Applications 

While all work in A I is ultimately founded on the basic elements of 
representation and process, the field has a number of traditional applications 
areas. Robotics is a basic applications area, involving planning and executing 
physical motions. Vision is another, involving encoding visual scenes into 
meaningful and useful representations. Another applications area involves 
language, both understanding and generating written and spoken natural language 
y.Vf. En ^ , | sn F r<*nch, etc.). Understanding spoken language is an extremely 
difficult problem that has served as an excellent test bed for AI ideas, but the 
problem is far from solved. 



The development of systems that learn from experience, can reason by 
analogy, and can recognize and manage their own resources and limitations are 
also important areas of AI research. The development of AI tools is naturally a 
field of intense interest. Software tools include programming languages, 
environments, and special purpose, high level systems. Related to software are AI 
efforts to generate computer programs automatically from English language-like 
specifications. Hardware is also a tooi. In the last few years, special purpose 
LISP machines have become commercially available on which to do AI 
development work. Further work on parallel processing hardware systems is an AI 
research priority. 

One AI applications area that just recently gained wide recognition is 
expert systems. The growth of AI R&D in the next few years will almost 
certainly be dominated by this applications field. An expert system is an A I 
program that exhibits or aids professional level human expertise in such areas as 
medical diagnosis, structural engineering, petroleum exploration, or computer 
system configuration. The development of expert systems has become a 
commercial enterprise, and to the extent that these systems do aid humans in the 
work they must do, expert systems will proliferate. 

Expert systems have become successful outside the research laboratory, 
prior to oth r AI applications fields, because thest >ystems work in very narrowly 
constrained environments. For example, the vocabulary and nature of discourse in 
medical diagnosis are tightly circumscribed in comparison to the vocabulary and 
discourse that appear in the newspaper every day. The vocabularies associated 
with professional activities provide a way to begin modeling the expertise 
involved. But with vision or speech understanding, the knowledge of how these 
things are done is much less developed. 

Developing expert systems, an activity known as "knowledge 
engineering," is, even so, not an easy task. For most genuine expertise, even 
though there is a vocabulary to describe what is done, there is no way to state 
how performance is achieved. People know more than they are aware of knowing. 
Because the details of expertise are beyond current understanding, knowledge 
engineering is as much the psychological science of unraveling the mysteries of 
intelligent human performance as it is the computing science of creating software 
exhibiting it. 

Summary . What AI field is most appropriate to applications in training, 
performance assessment or job aiding? The answer lies in the nature of the 
problem. The subject under discussion is technical expertise, and the AI field of 
expert systems has the most to offer. Expert systems are new, and in many ways 
fragile. Much work needs to be done on increasing their robustness. But, these 
systems have demonstrated their worth. This is not yet the case for natural 
language interfaces or vision systems. Vision and language interfaces will 
enhance the utility of expert systems, but it is the representation and utilization 
of technical knowledge that shall always remai.i at the core of what A I has to 
offer the operations and maintenance of Air Force systemr. 
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AI Literature 



The Handbook of Artificial Intelligence (Barr A: Feigenbaum, 1982) is a 
basic source of information on AL The handbook "is intended to be an 
intrcductory reference book Loth for researchers within AI and for those outside 
who need a guide tc the unchartered waters of AI research and literature" (Price, 
1982, p. 2). It is not a textbook, collection of separately authored papers, or a 
synthesis of the field. The handbook itself has a section about the AI literature, 
which is recapitulated here. 

The AI literature is generally categorized chronologically: early 
materials - before 1965, middle materials - from 1965-1975, recent materials - 
1975 to present. 

Ear ly publications. Books published during the early period include 
Computers and Thought (Feigenbaum tc Feldmen, 1963; and Semantic Information 
Processing (Minsky, 1968). A classic book of this period is Newell and Simon's 
(1972) Human Problem Solving, which has been heralded as the beginning of the 
discipline of cognitive science, the synthesis of the fields of artificial intelligence 
and cognitive psychology. 

Middle publications. Books of the middle period (1966-1975) include 
Understanding Natural Language (Winograd, 1972), Computer Models of Thought 
and Langua ge (Schank & Colby. 1973), The Psychology of Computer Vision 
(Winston, 1975), and Representation and Understanding (Bobrow & Collins, 1975). 
A general introduction to the field of AI during this period includes Sciences o f 
the Artificial (Simon. 1969). 

Recent publications . General AI introductions during this period include 
Godel, Eschew Bach; An Eternal Golden Braid (Hofstadter, 1979) and books by 
Norman and Rumeihart (1975) and Schank and Abelson (1977), which develop from 
first principles systems for natural language understanding. Textbooks about AI 
include Winston's (1977) Artificia 1 Intelligence, and Nilsson's (1980) Principles of 
Artificial Intelligence . Books dealing with LISP programming inci te Charniak, 
Riesbeck, and McDermott's o ??.0) Artifical Intelligence Programming, and Schank 
and Riesbeck's (J 981) Inside Computer Understands- Five Programs Plus 
Miniatures. 

Current publications . Books inj>ress include Winograd's Language as a 
Cognitive Process, and The Psychology of Human-Computer Interaction (Card. 
Moran & Newell). 

Very recently a number of excellent reference sources have been 
published concerning expert systems. Two books on this subject are Knowledge - 
Based Systems in Artificial Intelligence (Davis & Lenat, 1923), and Buildi ty 
Expert Systems (Hayes-Roth, 1983). Additionally, there are the following 
technical reports and journal articles: 'The Organization of Expert Systems, A 
Tutorial" (Stefik et al., 1982), "An Investigation of Tools for Building Expert 
Systems" (Waterman & Hayes-Roth, 1982), and "An Overview ol Expert Systems" 
(Gevarter, 1982). 
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Journals, periodicals, and conference proceedines . The most current 1 
Information about work in AI is found in journals, periodicals, and coherence 
pru^U^gs. These journals and periodicals include Artificial Intelligenc e the 
journal of the Cognitive Science Society, Cognitive Science ; the Amerl can~5ournal 
of Computational Linguistics! the Sl OART Newsletter, published by the 
Association '^.Computing .Yiachlneryi the American Association for Artificial 
c72W!?«!« *$ . . Magazine: the International Journal of Man-Machine 
. . studies JUMM5) which also publishes a great number of papers directly relevant to 

the application of AI to instruction? and several publications of the Institute of ! 
Electronic and Electrical Engineers (IEEE). 

Associations, conferences, arid proceedings Include the biennial 
International Joint Conference on Artificial Intelligence (IJCAI), and the national 
meetings of the AAAL The Cognitive Science Society holds annual meetings as 
well. Carnegie Mellon University annually holds a meeting on cognitive science, 
and the National Institute of Education and the U.S. Department of Education 
periodically sponsor seminars addressing issues relevant to the application of AI to 
instruction. 

Bibliographies. The bibliographies of the three volumes of The Handbook 
7j of _ Artificial Intelligence referred to at the beginning of this chapter provide a 

& comprehensive key to the bulk of published literature on AL Each topical section 

# °* the handbook includes a section indicating important references pertinent to 

# that section. Of particular interest to the study of applications of AI to 
.<j instruction is thr handbook section on applications-oriented AI research in 
& education. 



CHAPTER II. CURRENT RELEVANT 
DoD RESEARCH AND DEVELOPMENT 



number of Department of Defense agencies are becoming involved in 
R&D relevant to the AI applications under consideration in this report. This RfcD 
Is summarized in Table 1, which lists the sponsoring agency, project title, and 
performing organl'ation if different from sponsoring agency and known. In 
Appendix A the detail of this list is expanded to include the names and telephone 
numbers of responsible individuals (of both sponsoring agency and performing 
organization), project durations, levels of funding, and pertinent sources of 
additional information. Narrative descriptions of relevant projects are integrated 
into the main body of tWi report. 

, M J 1 * R * D reported in Table 1 is not strictly limited to projects explicitly 
dentifietf with artificial intelligence. Also reported is R&D in the three areas of 
interest (training, performance measurement, and job performance aiding) which 
forms a technology base for applications. The research reported also spans the 
continuum from basic theoretical work in cognitive psychology to projects that 
have near-term or midterm applications. 



DoD Centers of Activity 



As can be seen from Table 1, DoD is sponsoring substantial R&D in the 
target A I applications fields. The intent of this section is to summarize, at the 
program level, what each of the involved agencies is doing and is planning to do. 



Office of Naval Research (ONR) 

, n . m . £ m »i° r center of relevant activity is the Office of Naval Research 
{ONR), Psychological Sciences Division, Personnel and Training Research 
Program. This program includes areas in theory-based personnel assessment, 
information processing abilities, instructional theory and advanced training 
systems, and cognitive processing. R&D in each of these areas is directly 
relevant to the target AI applications areas. In particular, "work on instructional 
theories Jocks to eventual application in generative knowledge-based, automated 
systems for training. . ." (Office of Naval Research, 1981, p. 1), 

Additionally, ONR and the Army Research Institute are jointly funding a 
program in intelligent computer-based instruction (ICAI). This 3-year program, 
running from Ff 82 to FY 8;, has a total program funding level of about $2.5 
million. 



With respect to the future, program planning at ONR is looking foward 
to the funding of a center of excellence for ICAI. The goal of this center would 
be the development of A I tools for the creation and delivery of all types o* 
instructional material, both print and interactive. 



Table I. Current Relevant DoD Research and Development Projects by Sponsoring Agency 



Sponsoring Agency 



ONR* 

ONR 

ONR 
ONR 

ONR 

ONR 

ONR 
ONR 
ONR 

ONR 
ONR 
ONR 
ONR 



Title 



Performing Organization 



Analysis of Acquiring Strategic Knowledge in Problem Solving 

Cognitive and Instructional Factors in the Acquisition and 
Maintenance of Skills 

Cognitive Processes in Planning and Control 

Comprehension and Analysis of Information in Text and 
Pictorial Materials 

Computational Models of Problem Solving Skill Acquisition 

Computer-Based Modeling and Diagnostic Testing of Problem- 
Solving Skills 

Computer-Based Training of Mission Planning and Control Skills 
Human Understanding of Complex Systems 

Intelligent Automated Tutors for Instruction in Planning and 
and Computer Programming 

Intelligent Instructional Methods for Teaching Procedural Skills 

Mental Representation of Circuit Diagrams 

A Model for Procedural Learning 

Modeling Student Acquisition of Problem Solving Skills 



L'niverlsty of Pittsburgh 
University of Pittsburgh 

Rand Corporation 
University of Colorado 



Xerox Palo Alto Research 
Center 

Carnegie-Mellon University 



Percept ronics, Inc. 

Bolt Beranek 6c Newman, Inc. 

Texas Instruments, Inc. 

Bolt Beranek & Newman, Inc. 
Perceptronics, Inc. 
Carnegie-Mellon University 
Rutgers University 



•Office of Naval Research 
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Table 1 (cont.) 



Sponsoring Aconcy 


Title 


Performing Organization 


HMD 


Procedural-Net Theories of Human Planning and Control 


Science Applications, Inc. 


ONR 


TnC Rolf* of Prior Knnu/l«^0» in OnAritUn ^uiaa* 
"* vi • i iui r\iiuwic\igc An x/pcrallng UCV1CCS 


University of Arizona 


ONR 


Simulation Models frr Procedural Learning and Expert Memory 


Carnegie-Mellon University 


ONR 


Structural Apprcacl es to Procedural Learning from Technical 
Text and Graphics 


Cornell University 


ONR 


Studies of Skilled Human Performance 


University of California, 
San Diego 




Topic and Theme Identification In Prose Comprehension 


University of Arizona 


ONR 


Tutoring and Problem-Solving Strategies in Intelligent Computer- 


Stanford University 


ONR 


unaersianaing ana executing Instructions 


Bolt Beranek & Newman, Inc. 


NrRDC* 


Advanced Computer-Based Training for Propulsion and Problem 
Solving: STEAMER 


Bolt Beranek <Sc Newman, Inc. 


NPRDC 


Authoring Instructional Materials (AIM) 


NPRDC 


NPRDC 


Class "A" Electronic Equipment Maintenance Training System 


? 


NPRDC 


Computer-Based Techniques for Training Procedural Skiiis 


? 


4 NPRDC 


Theory of Graphic Representation 


? 



'Navy Personnel Research and Development Center 
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Table 1 (cont.) 




Soorisofinff Avencv 


TitJe 


Performing Organization 


NTEC* 


Artuiaal Intelligence Analyses 


University of Central Florida 


NTEC 


Automatic Knowledge Acquisition Techniques for Fxpert Systems 


Perceptronics 


NTEC 


Computer Aided System for Development of Aircrew Training 
canan cement 


Rowland and Company 


NTEC 


Personal Electronic Aid for Maintenance 


Texas Instrument j, Inc. 


NTEC 


Voice Te-hnology as the Instructor's Assistant 


Honeywell 


TAEG** 


Computer-Aided Authoring System 


University of Central Florida 


TAEG 


Training Aids Developed from Learning Guidelines 


TAEG 


AFHRL 


Artificial Intelligence in Maintenance Diagnostics and 

Information Systems 


Systems Exoloration. Inc. 


A CUD 1 


Automated Performance System C-3 Aircraft 


? 


AFHRL 


Computer-Based Maintenance Aids for Technicians 


? 


AFH"L 


Development of Draft Military Specifications for Maintenance 
Task Analysis and Logic Tree Troubleshooting Aids 


? 


AFHRL 


Evaluation of Individual Performance in Mechanical Specialties 


? 


AFHRL 


Interactive Computer Graphics Simulation frw intormotWiat* 
Level Maintenance Trainer 


Canyon Research Croup, Inc. 
and Denver Research 
Institute 


•Naval Training Equipment Center 

** Training Analysis and Evaluation Group 
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Table 1 (cont.) 

# 




Sponsoring Agency j^j e 


Performing Ora»ni*»tI/v\ 


AFHRL Plan for Application of Artificial Intelligence Technology 

to Technical Training, Performance Measurement, and Job 
Performance Aiding 


Denver Research Institute 


AFHRL Task-Oriented Measurement Technologies 


? 


AFHRL Unified Data Base Technology 


? 


ARI * Adaptive Computerized Systems (ACTS) 


Perceptronics, Inc. 


ARI Advanced Instructional Development: Intelligent Maintenance 

System 


? 


ARI Artificial Intelligence-Based Maintenance Tutor 


project in preparation 


ARI Intelligent Tools for CAI 


Yale University 


AFOSR* * Memory-Based Expert Systems 


Yale University 


AFOSR Systems Automation through Artificial Intelligence 




NAVAIRENGCEN Technical and Engineering Services to Determine Automatic Test 

Equipment Technological Use of AI Applications 


NAVAIRENGCEN 


•Army Research Institute 

**Air Force Office of Scientific Research 

f Naval Air Engineering Center 
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Table 1 (-oncluded) 




Sponsoring Agency 


Title 


Performing Orffanizatinn 


(joint funding) 






ONR/ARI 
ONR/ARI 
ONR/ARI 
ONR/ARI 


Computer-Based Program Consultant for Introductory Programming 
Computer-Based Tutoring for Diagnostic Skill Acquisition 
General Patterns of Scientific Inference 

intelligent Computer-Assisted Instruction: An Interactive 
Reading Tutor 


Yale University 
Stanford University 
Carnegie-Mellon University 
Yale University 


ONR/ARI 


Intelligent Computer-Based Instruction for Simulation Training 
of Troubleshooting Skills 


Bolt Beranek & Newman, Inc. 


ONR/ARI 


Principles of Intelligent Computer-Based Instruction for 
Cognitive Skills 


University of Pittsburgh 
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ONR is planning to start applied work in intelligent maintenance training 
simulation and intelligent job aids. This program will focus on the principles of 
human machine intelligence for computer-based training and job aiding. Finally, 
an effort is contemplated to combine R&D in several funding categories in oroer 
to bring together a coordinated effort bearing on the following areas: (1) adult 
basic skills, (2) technical training with an emphasis on computer-assisted 
instruction and intelligent maintenance simulations, (3) problem solving aspects of 
tactical training, and «0 the development of instructional materials, both print 
and interactive. 



tjayy Personnel Research and Development Center (NPRDC) 

NPRDC has three program areas specifically relevant to this report: 
personnel and occupational measurement, instructional technology, and training 
systems. Relevant projects underway at NPRDC are listed in Table 1. A project 
with the great direct relevance to this report is "Project STEAMER— Advanced 
Computer-Based Training for Propulsion and Problem Solving." 

Tiws objective of this project is to develop an advanced computer-based 
instruction system for a steam propulsion plant, including a detailed math model 
of a propulsion system, a graphics interface, and teaching strategies. The 
complete STEAMER math model with graphics interface has been implemented on 
development LISP machines. Development of a microcomputer-based STEAMER 
training system is underway. STEAMER is a long-term, high Investment project 
which has potential for adding considerably to the current state of the art of 
intelligent computer-assisted instruction. 

Another project worth noting is the Author' .g Instructional Materials 
(AIM) project. This has great commonality with the FY 85 ONR thrust area in the 
development of instructional materials. It is expected that these two efforts will 
be tightly coordinated. 



Navy Training Equipment Center (NTEC) 

AI anplications in training are being pursued at NTEC. Here, emphasis is 
being placed on tackling some of the problems associated with team training. 
Specifically, the concept of Al-based "surrogates" is being developed. For 
example, consider a surrogate opponent. In one-on-one submarine warfare 
training simulations, a simulated knowledgeable opponent could enhance the 
effectiveness of training by freeing the instructors from running the offensive 
submarine to perform other instructional roles. 

The surrogate team member concept is also being explored. When all 
members of a team are novices, the learning rate for the team as a whole can be 
slow. Having an individual trainee join a "surrogate team" (which already has 
competence) can accelerate the trainee's rate of skill acquisition. 

In addition to the surrogate concept, AI applications at NTEC are 
focused on performance assessment. "Assessment" should be interpreted as going 
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beyond the mere collection of data to the analysis and use of performance data in 
providing instructional direction and feedback to trainees. 



;3 AI applications in job performance aiding will most naturally integrate 

,< with ongoing advances in 3PA technology. Of note is the Personal Electronic Aid 

;< for Maintenance (PEAM) and other miniaturized, interactive job aiding devices 

under investigation at NTEC. 



JJ As at ONR and NPRDC, work focusing on the instructional design and 

a development process is also being pursued at NTEC, in particular, the computer- 

3 based system, CASDAT (Computer Aided System for Development of Aircrew 

3 Training). 



Training Analysis and Evaluation Group (TAEG) 

Work in the area of computer-aided authoring of inst: actional materials 
^ is also underway at the TAEG, which is located on the same site as NTHC Work 

£ is targeted on automated development strategies for the following kinds of tasks: 

j procedural, paired associate, and visual object classification. 

) TAEG Is also involved with the Naval Technicai Information Presentation 

' Program, a 5-year, muiti million dollar program to improve the Navy's efficiency 

in publishing. This effort is managed by the David W. Taylor Naval Ship Research 
: and Development Center and is now near completion. 

I 

\ Army Research Institute (ARI) 



3 The joint ICAI projects with ONR have already been mentioned. ARI has 

funded the Adaptive Computerized Training System (ACTS), designed to train 
electronics troubleshooting procedures. Beyond this, ARI, as of this writing, has 
no funded projects in AI relevant to the target areas. However, in preparation is 
a project termed Al-basrd Maintenance Tutor (AIMT) which is to function as tutor 
i and job aid. 



| Air Force Human Resources Laboratory (AFHRL) 

The AFHRL R&D program i? divided among four thrust areas: (a) 
staffing and force management, (b) air combat tactics and training, (c^ 
maintenance and combat support, and (d) training design and delivery. AI 
applications in training, performance measurement, and job aiding are potentially 
relevant to e*ch of these thrust areas. For example, if the scope of "performance 
measurement 11 is expanded to include personnel actions to be taken based on such 
measurements much of R&D going on in the manpower and force management 
area contribuves a technology base for potential AI applications. In this R&D 
area, sample projects include task-oriented measurement technologies; evaluation 
of individual performance in mechanical specialties; and the development of a 
prototype, computer-based training decisions system. 
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Ongoing research in air combat tactics and training also provides 
opportunity for potential AI applications, in particular, a desk-top trainer 
demonstration for utilization in pilot training and microcomputer-based special 
function trainer. A project focused on automated performance measurement for 
the C-5 aircraft offers the basis for extension of "measurement systems" to 
"assessment systems," where data are not only collected, but are analyzed and 
used to provide instructional feedback. There is apparent potential for 
collaborative efforts with NTEC in this area. As indicated in the introduction to 
this paper, AFHRL is planning a program of AI research, development, and 
application. 

The "surrogate" concept is also applicable to team training for operators 
of command, control, and communications systems. Another project that builds 
toward potential AI applications concerns the development of draft military 
specifications for maintenance task analysis and logic tree troubleshooting aids. 
If tlds guidance document were implemented as an intelligent, interactive 
development system, many potential advantages to AI applications would result. 
A project with similar relevance concerns the development of computer-based 
maintenance aids for technicians. Much work in the field of maintenance 
simulation affords opportunity for AI applications because such simulations can 
help shift the focus of simulation from the equipment to the task, where 
automated instructional intervention can be provided in response to student 
performance data. Project data for the efforts mentioned above are presented in 
both Table 1 and in Appendix A. 

As indicated in the introduction to this paper, AFHRL is planning a 
program of AI research, development and application. This paper addresses AI 
opportunities in training, performance measurement and job performance aiding. 
A parallel program planning effort is focusing on applications cf AI in the areas of 
maintenance diagnostics, intelligent maintenance information systems and in 
built-in test and automatic test equipment. These two parallel efforts are being 
coordinateo because it is realized that engineering design, maintenance, logistics, 
and training all are closely related and interacting subcomponents of effective 
and ready weapons systems. 



nter-Agency Coordination 

AI is new, and applications efforts even newer. Therefore, formal 
mechanisms for inter-agency coordination are not yet fully developed and further 
work on coordination is needed. Two obvious justifications of- such coordination 
are (a) the problems faced by the three armed services in training, performance 
assessment and job performance aiding are similar and amenable to similar 
solutions, and (b) the resource pool of trained people to do AI applications work is 
severely limited. 

A good start toward coordinating AI applications in training and 
maintenance is the Working Group in Simulation, Training and Maintenance of the 
Panel on Technology Initiatives in Artificial Intelligence, one of seven panels of 
the Joint Directors of Laboratories (JDL) Panel, a subgroup of the Joint Logistics 
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Commanders. The charter of the JDL is to put together joint service projects. 
Each of its seven panels has four working groups: electronic warfare; robotics; 
command, control, and communications; and simulation and training. Therefore, 
there is in existence an interservice coordination committee that is directly 
relevant to the coordination of Al applications in the target areas. The current 
working group consists of representatives from ARI, NTEC, NPRDC, and AFHRU 
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CHAPTER III. CHALLENGES AND APPROACHES 



Introduction 



The three training issues of concern to this study, training, performance 
assessment, and job performance aids (JPAs) are actually integrated pieces of the 
total Air Force training model. In the simplest description of the relationship 
among the three elements, training provides the basis for baseline job 
performance expertise, which is in turn augmented and developed through on-the- 
job training (COT). The effectiveness of either form of Air Force training is 
assessed through some form of performance measurement. The JPAs, notably 
technical orders, support the knowledge and skills of personnel by providing 
procedural aids for operations and maintenance tasks. Numerous trade-offs are 
required among (a) which knowledge and skills should be taught formally in 
residence training, (b) which should be acquired through OJT and experience, and 
(c) which need not be specifically taught because TOs and other JPAs are 
available. It is necessary to look at the entire picture both instructional^ and 
institutionally to arrive at a reasonable set of compromises concerning trade-offs. 

A common denominator among the three areas of interest is task 
analysis; it is the basis for training, evaluation, and JPAs, Therefore, task 
analysis is singled out as a separate issue area in which to corsider the potential 
of AI applications. 

Since training, measurement, and job performance are embedded in a 
larger organizational matrix, a systems analysis is appropriate in analyzing the 
challenges and opportunities in these areas. Two of the most important 
consideraticns from the systems point-of-view are the Issues of validation and 
system maintenance. 

Validation is centrally important in training, performance measurement, 
and JPAs. Often the most effective validation paradigms involve crossing the 
boundaries imong these three fields, as when a training program is evaluated in 
terms of the on-the-job performance of its trainees. 

An issue related to validation is that of maintenance. For example, as 
field evaluations show training to be deficient in certain areas, the training 
programs must be maintained to accommodate these deficiencies. If a program of 
applications of AI technologies were to focus solely on training, individual 
assessment, and JPAs, then important and consequential opportunities for the 
application of AI technology would be overlooked; the ultimate payoff of 
applications in any of the three basic areas might ultimately be discounted by 
forces operating on the larger systems level. 

This section is organized around challenges and opportunities related to 
task analysis, training, individual assessment, job performance aiding, and the 
integration of these areas into one system. Within each of these areas, a brief 
description of relevant challenges is devsloped, followed by a discussion of 
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potential approaches to the application of AI in meeting these challenges. 
Related past, current, and planned DoD R&D in each area is integrated into the 
discussion of approaches. 



Methodology 

Information regarding the challenges facing the Air Force training, 
assessment, and job performance support communities has been gathered from 
various sources. These sources include informal briefings from AFHRL personnel, 
input from consultants (Dr. Peter G. Poison, Department of Psychology, 
University of Colorado; Di. Mark L. Miiler, Vice President, Computer*Thought 
Corporation, Dallas, Texas), the technical literature, and discussions with DoD 
and civilian professionals working in these areas. 

Sources of information regarding approaches to meeting the challenges 
identified include: the past, current, and planned research activities in the 
application of AI to these areas. Other sources are advice from consultants 
regarding the more and less feasible approaches to specific problems and a 
general synthesis of the potential for application of AI technology to the 
challenges identified. 



Task Analysis 



Task analysis is the common denominator of training, individual 
assessment, and job performance aiding. As embodied in the Tri-Service 
Instructional Systems Development (ISD) approach, task analysis, in the form of a 
set of objectives^ is the basis for the development of both instructional materials 
and criterion referenced tests of mastery. Task analysis also plays an important 
role in performance assessment, as well as in the development of job performance 
aids. 

Naturally, the exact definitions of task analysis, as well as the 
methodologies and procedures for carrying them out, vary from training, to 
assessment, to job performance aiding environments. For example, within the 
training environment, an analysis of the prerequisites to new objectives and 
desirable sequencing of learning events may be considered a part of task analysis 
(Gagne, 1977). However, in the fields of performance measurement and JPAs, 
prerequisites and sequencing are less relevant. 

Setting such differences aside, common to training, assessment, and 
performance support is the need to identify the specific behavioral and 
information processing actions a human must accomplish in effective and 
efficient performance of a task or job. Because of this commonality, application 
of AI technology to the challenges of task analysis will benefit all three areas 
individually. 
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Challenges 

What are th<. challenges facing task analysis? Successful solutions to the 
various training and maintenance problems in the Air Force require a correct 
description of the kinds of knowledge that are necessary to carry out the various 
tasks and the development of a training program that in fact imparts this 
knowledge. Informal analyses of these training requirements, while insightful and 
often correct, lack the rigor and the detail necessary to deve'.op training programs 
that are independent of the expertise of a given individual. Informal analyses 
(even when conducted by very competent individual*) often turn out to be 
glaringly incomplete because gaps (which must be accomplished by the novice) are 
filled in implicitly by the set of experts who prepare the training material. They 
unJerstand the task, they write a description that seems reasonable to them, but 
since they already know how to accomplish the t<sk, they have no way of 
evaluating themselves to see if all the steps to accomplishing the task ar ; 
explicit. 

As a remedy for this, an adequate validation can be carried out as a part 
of the ISD cycle, but this feedback loop is slow because the effects of the training 
courses must be observed on-the-job and reported back to Air Training Command 
(A rC) before adjustments to the training can be considered. 

Another task analysis problem that impacts both training and assessment 
is the difficulty of developing a task analysis that is generaiizabie. For example, 
a survey of B-52 refueling procedures identified seven different techniques at 
seven different bases-all technical!/ correct. What is common among thes* 
techniques? What motivates the discrimination among them? What should be 
taught, given that teaching all seven different possibilities is infeasibie? What 
standard should be employed in assessing performance of this task? 

V Task analysis, is critically relevan: in the L.ea of performance 

. assessment. A major problem in this area is identifying what is to be measured. 
.Often what is measured is based on an analysis of the job by one isolated content 
^expert. According to some observers, this approach has failed for the Army in its 
^kili Qualification Test program. A critical qut tion is how to get a 
riyresentative decomposition of a task. How many expen performances must be 
analyzed before a sufficiently broad -base description of peiformance is in hand? 
In vbe author's opinion, the Air Force Occupational Survey Data do not break down 
tasks, into enough detail to be a solution to this problem. 

\ Tne problem of task analysis is not confined to differences in task 
approaches within one task area. There is a need to identify subtasks common 
among sets of tasks, so that training efficiency can be maximized through a 
proper clustering and sequencing of training objectives. The Air Force needs to 
have an accurate model of the expertise in its technical corps on which to base 
manning decisions. The OJT performance records must be more descriptive, 
recording more than limited, simple numeric scores. A tracking system is needed 
that can support realistic and effective deployment of technical expertise. Such a 
system must be based on detailed task analyses. 
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Approaches 



Some tasks are easier to analyze than others. At one end of a continuum 
are jobs or tasks that are completely routine, fixed-sequence procedures, 
incorporating known actions and methods in the service of predetermined, specific 
goals. Normal disassembly of a piece of equipment would fall In this category; so 
would using a word processor to edit a manuscript. At the other extreme are 
tasks for which it is difficult or impossible to specify the goal, component actions, 
and methods. Many management jobs fall at this extreme of the continuum; so do 
many troubleshooting and problem solving tasks such as fault isolation in digital 
equipment or writing and debugging , computer programs. Let us call those 
simpler, easier tasks, routine tasks, and the complex, difficult to define tasks, 
genuine problem solving tasks. Approaches to the challenges of task analysis 
differ, depending on whether the tasks are routine or are genuine problem solving. 

Routine tasks. Kleras and Poison (1982a, 1982b) have developed a 
theoretical approach to the formal analysis of task complexity in routine 
cognitive tasks. The tasks of particular interest to their project deal with 
Interactive computing. Their approach is built on nrid extends previous work in 
cognitive psychology, cognitive science, and AI. It is a relevant synthesis and 
application of prior work in these fields to task analysis. 

The approach Involves the development of a formal description of the 
user's knowledge of how to use an interactive device and a formal description 
system for representing the interactive device Itself. The user's knowledge of the 
task and the device are represented using formal representation schemes from AI. 
Details of both of these representations are presented in Kieras and Poison 
(1982b). The formal descriptions are computer simulation programs. These two 
descriptions are validated by running one against the other. In other words, the 
/'armal description of the user's knowledge (e.g., the task representation) is 
validated by seeing if it can interact, in the service of some useful goal (e 
editing a text), with the formal description of the device. If either description is 
incomplete or inaccurate, the two simulations will not properly mesh; that is, they 
will not run, or they will produce erroneous results. This approach is appropriate 
only for routine cognitive skills— tnose for which it is possible to completely 
specify a running simulation of the task. Routine cognitive tasks are certainly 
required in many career specialties for which this approach, without modification 
Ci extension is applicable. Even when jobs do contain genuine problem solving 
components, the routine tasks can be singled out and treated with the Kieras and 
Poison f ormalism. 

Given the \wo models of the Kieras and Poison formalism, a number of 
metrics of task difficulty can be derived. One way that task difficulty can be 
interpreted is by the size of the how-to-do-it knowledge representation. Another 
way is to count the depth of the number oi subgoals that need tc be invoked to 
accomplish that task. Once these measures of task difficulty are known, 
Instructional resources can be allocated in proportion to the difficulty of the 
tasks. Moreover, the task representation for any subtask gives a precise 
specification of exactly what must be learned in order to perform that task. Such 
metrics not only help in deciding where to place instructional emphasis, they also 
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help in deciding exactly what it is that must be taught. In fact, It may be fairly 
straightforward to translate the task representation into instructional text or 
other media. 

The metrics are of similar use in evaluation. Through exploration with 
simulations, it would be possible to develop a criterion task that exercises an 
arbitrary proportion of the task hierarchy. Hence, efficient and effective 
performance measurement tests can be designed. Also, since technicians may 
often be the least competent on those tasks which require -he most learning, such 
tasks can be placed under special scrutiny, especially if they are critical to the 
job. 

Another outcome from the Kieras and Poison approach is the knowledge 
that follows from a mapping of the task representation to the device 
representation. If there is a mismatch between these two, there probably will be 
difficulty in learning the correct procedures in the areas of the mismatch. This 
can be corrected by either redefining the way the task is represented (re-doing 
the instruction), or by redesigning the device itself. In this manner, the man- 
machine interface can be optimized before a device is developed. In a similar 
vein, since task complexity is "induced" by the design and structure of the device 
(as represented by the device representation), prior to prototype development of a 
device its representation can be manipulated, observing the effect on the induced 
task complexity. The goal of such an exercise would be to settle on a device 
design which produced the minimum human operator task complexity. 

In addition to the task and device representations, an important 
representation is of the user's understanding of "how-it-works" knowledge for a 
device. The knowledge contained in a mental model of a device is a source of 
meaningful explanations of an individual's procedural knowledge, the task 
representation. Such mental models help novices' initial attempts to iearn how to 
operate a device. The how-it-works representation is a hierarchy of explanations. 
Kieras and Poison (1982a) stipulate several criteria for selecting those 
explanatibns from the how-it-vvorks hierarchy that should be selected for 
presentation to a novice. 

In sum, tf-» Kieras and Poison formalism provides a method for 
developing and validating task analyses of routine cognitive tasks that can be used 
(a) to Hirect the design of new devices, (b) to identify subtasks that will be 
difficult to learn, (c) to specify what must be learned, and (d) to specify what 
how-it-works knowledge would be useful explanation to novices. 

Genuine problem solv ing tasks . For the Air Force the need to support 
training, measurement, and job performance is more critical for genuine problem 
solving than for routine tasks. In briefings by staff members at the AFHRL— 
Training Systems Division, avionics maintenance in particular was mentioned as in 
need of support. In spite of advances in automated test equipment, this job often 
requires genuine problem solving competency. Avionics maintenance is 
representative of a larger class of genuine problem solving tasks, namely, fault 
isolation, diagnosis, and troubleshooting tasks. There is a large amount of R&D 
experience -vith training and performance in these areas that could be pulled 
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together to form the foundation of AI applications effort in training, assessment, 
and Job performance aiding In these critical task areas. 

The task analysis of genuine problem solving tasks can be better 
understood by considering the work that has been accomplished by researchers in 
cognitive psychology and AL Large strides in the theoretical understanding of 
problem solving behavior have resulted from the synergistic relationship between 
these two fields. Work in this area has moved away from the domain of games 
(checkers, the Tower of Hanoi, chess, etc.) and has focused on genuine problems. 
Much work by psychologists* has focused on problem solving In domains such as 
algebra or physics. Work in areas of "real world" practical significance, combined 
with an emphasis more on performance than on theoretical development, results 
in the familiar "expert system." 

The methodologies and technique* used by both psychologists and 
"knowledge engineers" are of use to the Air Force in developing approaches to 
formal analysis of genuine problem wiving tasks. The one pre-emiront 
consideration is that in the knowledge about a certain field or domain lies most of 
tne Power to solve problems in that domain, 

» 

Psychologists have developed a modern cognitive theory of problem 
solving in the tradition of information processing psychology (Chi 6c G laser, 1979; 
Greeno, 1976a, 1976b, 1977; Hinsley, Hayes & Simon, 1976; Larkin, 1979a, 1979b, 
1980a; Larkin, McDermott, Simon & Simon, 1979a, 1979b; Mayer, 19?*, 1973, 
1980; Mayer & Greeno, 1972; Mayer, Greeno & Stiehi, 1975; Newell 6: Simon, 
1972). Problem solving, to this tradition, Is characterized as understanding and 
search (Poison & Jeffries, 1980). Expert "understanding" in physics problem 
solving, for example, involves the use of a qualitative scientific representation of 
the phenomena at hand as a guide to the selection of a method of solution (Chi, 
Feltovich Ac Glaser, 1980; Greeno, 1980; Larkin, 1980b; Richardson, 1981). 
"Search," for the expert, has the character of building toward a goal, rather than 
working backwards from a goal. 

A problem solver (expert or novice) wlren faced with a problem, has a 
representation in memory to which he or she fits the problem. Depending on the 
degree of meaningfulness of that fit, the solver then brings to beoi' a number of 
search processes that act on the representation, filling out a path to a solution. 
Depending upon the degree of meaningfuiriess inherent in the representation to 
•which the solver fits the problem, the search may be straightforward or it may be 
circuitous; a solution may result forthwith or it may elude the solver entirely. 
Often a problem representation must be adjusted in the course of developing a 
solution, for example, from a functional representation to a physical 
representation, or from a conceptual scientific one to a mathematical one. 

It is important to note that the representation (also termed cognitive 
structure or schema) brougnt to boar on the problem is an organized collection of 
experience with this and similar problems. Thus, problem solving is knowledge- 
dependent or knowledge-based. Skill in problem solving is not reducible to a 
general procedural skill alone. Rather, it is in part determined by a large body of 
organized knowledge structured for the purposes of solving problems In a 
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particular domain and in part determined by a set of quite specific rules, 
procedures, or heuristics which act upon that knowledge. 

AI research a'«o has much to contribute to understanding genuine 
problem solving tasks, in particular, expert systems technology. Consider those 
expert systems in use today based on a technique known as backward chaining. 
Basically, expertise is captured by a set of rules of the form: if some condition(s), 
then take this action. In backward chaining, the initial conditions of a goal 
statement are examined, and rules whose actions will satisfy those conditions are 
•^ftJ'-i Tl 2 is l*oce** is repeated recursively until all of the goal condition Is 
satisfied. An explanation of this process is provided by Waltz (1982). A design 
goal for such expert systems is to have the set of rules which are comprehensible 
to human experts. In this way, the human expert can "understand" how the expert 
system arrived at a particular conclusion or answer by examining the chain of 
rules which led to that answer. 

MYCIN (Shortliffe, 1976) is an example of a backward chaining expert 
system. The backward chaining control structure is implemented through the 
interaction of two mechanisms, MONITOR and FINDOUT. MONITOR'S function is 
to verify whether or not a rule is true. To do thb, it must establish the truth 
value of each of its conditions. If one of the conditions is not true, the rule fails 
and its actions may not be taken. Often it is not possible to determine the truth 
2fjjg«* one of the conditions. In such cases the FINDOUT mechanism Is called. 
FINDOUT retrieves the list of rules which have an action which establishes the 
truth value of the condition in question. At this point, the problem is to see if any 
of these rules are true. In order to do this, FINDOUT calls MONITOR with the 
task of examining each of these rules. If a condition in one of the** rules is 
indeterminate, the process is continued working backward (e.g., FINDOUT is 
called again). Backward chaining is recursive because of the way a mechanism 

ffJS.iHSJ in th 5 course of Processing (e.g., MONITOR calls FINDOUT which calls 
MONITOR, etc.). 

Backward chaining systems typically do not explicitly contah models of 
causality, functionality, or structure for the domain under consideration. This is a 
limitation because, for the toughest of problems, it is often necessary to reason 
from a causal, Junctional, or structural model of the domain in order to arrive at 
a solution. These and other issues regarding expert systems are cogently 
addressed in a paper by Randall Davis (1982). 

In seeking help for the challenges facing the Air Force in the 
development of valid and useful analyses of tasks that require genuine problem 
solving skil. , slight differences in orientation between the psychological and AI 
oriented app-oaches must be considered. In psychological models of problem 
solving, comprehensiveness, rigor, and theoretical value are of more importance 
than is performance capability. The opposite is true of AI expert systems, 
because performance is of paramount interest, and at times certain aspects of 
performance are forced or glossed over in ways that do )t parallel or model 
human performance, for example, in exhaustive search. Both of the approaches to 
genuine problem solving have value to the Air Force in enhancing the 
methodologies of task analysis in complex domains. The psychological approach is 
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J Important because the task analyses will be applied to Instruction; ultimately, 

human performance Is the subject of interest. On the other hand, in tlie area of 
JPAs, it may be permissible to have an expert system arrive at some conclusion or 
take some action (such as a curriculum sequencing decision) that is not entirely 
the way a human would do it 



I Training; In-Resldence and On-the-Job 

Challenges 

3 Training in the Air Force is provided both as formal instruction and on- 

i the-Job. By far the greatest formal training emphasis is on in-residence training. 

» For the Military Airlift Command and Strategic Air Command there is a trend 

toward less formal schooling, and more COT at the unit level. For ATC the trend 
Is toward more cenfra/ized training and more self-paced/CAI training (Rosenblum, 
1979). OJT relies almost exclusively en local unit management of training and 
certlf icatlon programs, largely on an informal and subjective basis. Challenges in 
j training include: 

j 1. The need for valid, quality, useful task analyses. 

j 2. The need to present instruction that is a proper blend of 

rote learning of procedures and robust learning of 
relevant principles. 

3. The need for efficient and timely methods of instructional 
materials development and revision. 

<». The need of effective instructional materials. 

3. The need for less reliance on actual equipment as the sole 
source of practical hands-on, job-relevant experience. 

Shortage of trained personnel is a good organizing principle for 
considering training issues. The ultimate source of shortage is personnel attrition. 
Training programs produce individuals with expertise that is required in the 
private sector? the Air Force often acts as a training ground for industry. In 
addition, a shortage of expertise results from the continuing evolution in the 
complexity of Air Force weapons systems. This generates a demand for oersonnel 
who have the skill to understand and work with systems that are increasingly 
complex and technical. 

This issue of expertise shortage is at the heart of a conflict between the 
training and field missions within the Air Force. The field feels a shortage of 
trained personnel and would like the graduates of the training programs to be 
trained to a higher technical caliber. But producing this caliber robs the field of 
expertise (in the form of instructors) and of useful duty time (in the form of extra 
time spent in training). The Air Force needs to maximize in-residence training 
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graduates' competence while minimizing the amount of resources spent. A 
solution lo this problem is to increase training effectiveness both in-residei>ce r nd 
especially 03T. Of course, this has lon^ been a goal for the Air Force, and 
progress toward this goal is continuously being made. 

The allocation of training objectives differentially to in-residence and 
OJT programs is currently done on a formal basi^. An A R HRL planned Training 
Decisions System is designed to act as a management aid in making these 
allocation decisions, decisions which have a large potential impact on overall 
training effectiveness and eificiency within the Air Force. 

AI techniques can provide approaches to increasing training 
effectiveness. The approaches, suggested below, may be applied to either the in- 
residence or on-the-job training environments, r^cause in-residence training 
emphasizes classroom sized groups of students, applications of instructional 
technology may be most appropriate In the 03T environment where training is 
more structured around the individual than in in-residence training. The needs 
identified previousdy are found in both in-residence and OJT environments, and 
approaches suggested in the following paragraphs, in appropriate mixes, will be 
suitable in both of these environments. 



Approaches 

Task analysis. AI applications in support of this foundational component 
of instructional systems design have already been treated in detail. Successful 
progress in areas of identified need in training depend on front-end work in task 
analysis. The task analysis dees not satisfy, in and of itself, any pressing training 
need. But, application and use of task analyses are of fundamental importance in 
instruction. In the remaining subsections, particular training applications of task 
analyses, the cognitive psychology of problem solving, and work in knowledge 
engineering and expert systems are presented. 

Selection of the ap pr opriate mix of rote learning and general principles 
learning. This is an old and familiar issue, and some have observed that the 
current orientation toward this issue swings from pole to pole cyclically. As old 
and timeworn as this issue is, recent advances in cognitive science provide fresh 
insights into the problem of rote versus meaningful instruction. 

The Kieras and Poison formalism previously discussed provides a 
straightforward approach to this problenr.. They have developed a set of criteria 
for deciding when "how«it-works" knowledge should be useful to novices. Briefly, 
these criteria are (a) if the user's task representation contains a goal that is 
unique to the device the user is working with and not previously encountered in 
cifferent devices, then that goal should be supported with an explanation derived 
from how-it-works knowledge, (b) if a goal in the task representation is already a 
part of the user's expected standard repertoire of performance capability, that 
goal need not be supported with explanation, and (c) if how-it-works knowledge 
does not correspond with a particular go^al in the user's task representation, then 
that explanation shouid not be provided./ These rules are familiar; the second one 
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addresses the entering behaviors consideration in instructionai design, and the 
third is a tenet ol functional context training (Shoemaker, I9o0). What the Kieras 
and Poison formalism contributes is a formal descriptive sys;em on which to base 
these rules. 

!' Much work has been done on meaningful versus rote learning, beginning 

tj with classic studies by Brownell (1935). More recently, the work of Mayer (Maye-, 

J, 1974} Mayer & Greeno, 1972} Mayer, Greeno & Stiehl, 1975) sheds light on the 

I Importance of this problem. The findings from these experiments on rote versus 

meaningful Instruction for a variety of tasks indicate that qualitative differences 
n learning outcome are found as a function of instructional treatment. During an 
Instructional treatment, students will selectively attend to particular aspects of 
the presentation In accordance with the learning "set H activated by the treatment. 
If students expeci Instruction to be rote, then they learn by rote. If they expect 
meaningful instruction, then they attend to meanings. The outcome of a 
meaningful approach Is that students are more capable of solving novel problems, 
but * ive familiar problems less quickly than do students trained with more 
emphasis on « ote learning of fixed procedures. 

Overall, work In the psychology of expertise yields the result that, once 
again, performance is largely a function of command of and access to large 
amounts of knowledge. It has been estimated that expertise in chess (and perhaps 
expertise in numerous fields) may be accounted for by some 50,000 "chunks" of 
knowledge which require upwards of 10 years to acquire. If this is the case, then 
even though meaningful, principles-oriented instruction promotes a certain 
robustness in learning and memory, there is no substitute for experience. The 
training implications of this finding are that in performance areas where expertise 
is needed (as in genuine problem solving tasks), one should not expect too much 
from training. If expertise is in critical shortage, JPAs may be a more realistic 
solution than beefed-up training programs. This is further treated in the section 
below on JPAs. 

One of the most interesting findings of the oroblem solving literature 
(cited above) is that novice and expert performance can differ qualitatively in 
reaching the correct solution to a given problem. The explanation for this lies in 
the different representations and search methods employed by novices versus 
experts. Misconceptions in student understanding (McCloskey, 1982} Stevens, 
Collins 4c Goldin, 1978) further contribute to the distinction between novice and 
expert. It has become clear that protocol analyses of novices (e.g., -**k analyses 
from the novice's point of view) are critically important in understanding how to 
approach instructor It is not sufficient to expect novices to learn \he expert 
approach. First, the misconceptions must be dismantled, and then a transitional, 
developmental progression from novice to expert must be supported with 
instruction. This realization is at the root of an approach to the use of an expert 
system as the basis of a tutorial system (Ciancey, 1979; Clancey tc Letsinger, 
198 i). The rules in an expert system are tightly "compiled" pieces of knowledge. 
They contain, implicitly, structure, and procedural knowledge that must be made 
available, explicitly, to the student. In addition, justifications for the rules must 
be avaiJabie to the student. Justifications serve as memory aids for remembering 
the rules and as a basis for deep understanding that allows students to learn 
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beyond what is captured in the rules. Expertise cannot be transmitted intact; a 
lot of recompilation ol basic principles, under the guidance J[ experience* is 
needed before fresh expertise can be developed. 

Building blocks and tools for conventional Instruction and CAI . NPRDC 
nas launched the Authoring Instructional Materials (AIM) project focused on using 
computers to automate and assist curriculum development for both computer- 
delivered and conventional instruction. TAEG has underway the Computer Aided 
Authoring System project with the objective of developing an authoring system 
capable of generating job aids, learning aids (procedural and nomenclature), 
curriculum, instructor guides, and student guides. 

There currently exists a number of authoring and/or curriculum 
development systems, building blocks, tools, or aids, for example, the Writer's 
Workbench at Bell Labs, the Computer Readability Editing System (CRES) at 
TAEG, the Computer- Aided System for Developmental Aircrew Training 
(CASDAT) at NTEC, Computerized Job Aids for ISD at ARI and the Human 
Resources Research Organization (HumRRO), student flow simulation for course 
design at the Air Force and Rand, and commercially available authoring systems 
for computer-based instruction available from such companies as WICAT, Bell & 
Howell, Control Data Corporation, Hazeltine, etc. 

An emphasis or reliance on isolated building blocks and tools for 
designing and developing instruction may not be the best approach to the problem 
of developing conventional CAI. A standard way of conceiving of computer-based 
instruction software systems is to see text and graphics (e.g., display-oriented 
editors) as a basic building block, which is encapsulated within an instructional 
Interaction editor used to build lessons (e.g., a CAI editor), which Is in turn 
encapsulated within a curriculum management editor controlling the progress of 
students through sets of lessons constituting courses (e.g., a CMI system). Many 
systems exist that are structured in this way, c.f.: WICAT, AIS, PLATO, etc. 
While these systems do provide structures to be filled in at various levels (course 
hierarchies, objectives, and instructional interactions), there is little or no help 
and assistance for the instructional designer in making design decisions, such as 
which objectives to select, how to structure and organize them, how to develop 
effective feedback, or how to build adaptive branching schemes. It is suggested 
that this is an area where AI might be applied. 

The effectiveness, efficiency, and timeliness of courseware design, 
development, and delivery can be increased by integrating these three "Ds" into 
one coordinated system. At AFHRL, the Interactive Graphics Simulation (ICS) 
project has in place an integrated design, development, and delivery system for 
task-oriented avionics maintenance simulations. Control Data Corporation is 
currently placing an emphasis on this approach. 

The key to the ICS approach lies in the interactive development, 
refinement, and modification of a hierarchical task analysis of the job. This is 
currently accomplished on-line on the AIS through use of specially designed 
editors. The resultant on-Jine task hierarchy is then used to develop simulation 
scenarios, complete with scene-setting text and graphics, video disc control 
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commands, and feedback messages. During delivery, the task hierarchy is used to 
represent the alternatives available to the problem solver at each step of the way 
towarda solution. Since correct and incorrect choices are nodes in the task 
hierarchy, a record of a given student's responses across a number of simulations 
provwes a model of the student's competence in terms of the task hierarchy. This 
model is then used to control instructional interaction toward the goal of focusing 
student attention on unmastered portions of the task. For example, when a 
student's response history indicates that the student has mastered a commonly 
reoccurrlng subtask, that subtask is summarized, thus freeing up time and 
attentlonal resources for subtasks that do need further practice. Explanations and 
lustrations of this integrated design, development, and delivery system are 
presented in Dailman (1982), Richardson (1982) and Pieper (1982). 

i ^ i The J* 6 * t0 success in developing such an integrated system was the 
pivotal use of an on-llie task analysis representation. As emphasized in earlier 
sections on task analysis, a task analyst approach to integrating systems can be 
extended beyond adaptive drill and practice (which is what the interactive 
graphics simulation system is) to include explanatory instruction, performance 
measurement, and job performance aiding as well. When systems are oosely 
integrated through the use of task analyses, this facilitates courseware 
maintenance and helps establish courseware validity. 

Intelligent CAL Intelligent CAI (ICAD offers a way of enhancing the 
effectiveness of interactive computer-based instruction beyond what is achievable 
with traditional approaches to tutorial, drill and practice, and simulation 
courseware. ICAI seeks to enulate the professional competence of good teachers 
when working one-on-one wit i a studen.. Good teachers (a) know what they are 
doing, ..e., have subject matter competence; (b) are vitally concerned with 
understanding the state of their students' knowledge, including misconceptions; 
and (c) have a good "bag of tricks" concerning pedagogical approaches, for 
example, the good teacher knows that the student needs a proper mix of 
instruction and practice. AI in general tries to emulate the actions of intelligent 
agents, and ICAI attempts to enuiate intelligent teachers. 

The main components of an ICAI system are problem solving expertise, 
the student model, and tutorial strategies. An exhaustive review of the ICAI 
lite, ature is beyond the scope of this paper. Integrative reviews of this field have 
appeared in the literature, and include those by Gable and Page (1980), Barr and 
Feigenbaum (1982), and Sleeman and Brown (1982). 

Table 2 briefly summarizes a number of ICAI systems, indicating 
literature references and subject matter nature. Not all of the th^ee components 
of an ICAI system are fully implemented in every system. Each component by 
itself is a difficult problem, and the successful implementation of a demonstration 
program often depends on strategic decisions regarding where to try to work at 
the state of the at t, and where to settle for well-known approaches. For example, 
the mere effort that has to be put into explicating domain expertise, the less 
resources are available for developing student modeling and pedagogical modules. 
ICAI systems consist of three main components, subprograms, or modules. That 
is, at the highest level of program organization, an ICAI «ystem 
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Table 2. Intelligent CAI Systems 



SYSTEM 


REFERENCES 


SUBJECT MATTER CONTENT 




uaroonell \ Yil\j) 
Collins (1976) 


South American geography, text 
editing systems 


WHY 


jveveiw, i^oii&ns oc VjOiuin 117/0/ 
Stevens 6c Collins (1978) 


extension of SCHOLAR 


SOPHIE 


orown, burton oc de Kleer (1982; 


electronics troubleshooting 


WEST 


Burton & Brown (1979) 


elementary arithmetic game 


WUMPUS 


Carr6c Goldstein (1977) 
Goldstein (1977, 1979) 
ovansziejuf i*arr oc vjOiostein 
(1976) 


fantasy dungeons and dragons game 


GUIDON 


Clancey (1979) 

v^iancey oc Letsinger u"8U 


infectious blood disease 


BUGGY 


Brown 6c Burton (1978) 
Brown 6c VanLehn (1980) 


elementary subtraction 


EXCHECK 


Blaine 6c Smith (1977) 
Smith, Graves, Blaine, 6c 
Marinov (1975) 
Smith 6c Blain(1976) 
Suppes (1981) 


logic, set theory, proof theory 


BIP 


Barr, Beard 6c Atkinson (1976) 


BASIC programming 


ACTS 


Hopf-Weichel et al. (1980) 


electronics troubleshooting 


STEAMER 


Stevens et al. 
(1982) 


propulsion engineering 


ADA TUTOR 


Computer* Thought Corp. (1982) 


Ada programming 
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consist* of an expert module, a student model module, and a tutor module. A very 
brief summary of the three ICAI components follows. 

The expert module. The expert module serves to generate and solve 
problems. The first role for this module is the generation of problems. This 
function is closely linked with the student model and pedagogical modules because 
curricular decisions involve what the student knows or does not know and 
strategic decisions about what to present next and how to present it. The degree 
to which ICAI systems generate curricula of problems in concert with student 
modeling and pedagogical considerations is very limited. All systems do have a 
scheme for sequencing problems, and approaches vary. The second role for the 
expertise module Is in solving problems. This is done in order to evaluate and 
critiqu* student solutions. ICAI systems with "articulate" problem solvers can 
explain how they reached a solution or how they would like the student to try and 
reach a solution. If the problem solver is not articulate, what is termed "opaque," 
then the system can offer no detailed critique of a student's solution path. The 
extent fc, which ICAI systems 1 problem solvers are articulate or opaque also 
varies. 

The student module. The objective of the student modeling module is to 
understand what curricular objectives the student has mastered, and to understand 
or have representations for predictable misconceptions and suboptimal 
approaches Input V the student model may be derived from numerous sources 
including: (a) a differential comparison of the student's behavior and the behavior 
output of the expert nodule on a given problem or question, (b) explicit 
information derived from direct questions asked of the student, and (c) historical 
assumptions based on th'.* student's experience. The student's knowledge can be 
represented as an overlay or subset of the expert's knowledge, or it can be 
represented as deviations from the expert's knowledge (e.g., misconceptions, 
suboptimal approaches, and "buggy" procedures). This is a very important 
distinction. As was explained in a previous section on the difference between rote 
and meaningful instruction, expertise is not something that can be transferred 
directly to a novice. There is a developmental process that must be gone through 
as the backward chaining approach typical of "uncompiled" novice competence is 
refined (through practice, experience, and more knowledge), into the pattern 
matching, methods application, forward reasoning character of expert 
performance. To represent and use a genuine expert representation for the expert 
in an ICAI system may be a mistake, as was discovered by Clancey in his 
experiments with M V CIN (Clancey 6c Letsinger, 1981). A deeper representation is 
needed in order that differences detected between the student's performance and 
that of the experts are to be minimized. The purposes or plans a student is using 
must be inferred from the student's actions. Therefore, "plan recognition" and 
plan representation are of great importance in ICAI systems R&D. 

The tutor module. The third module is the tutor module. It is a well- 
known fact that simply because someone has complete subject matter mastery, 
his or her ability to transmit that mastery does not necessarily follow. There is a 
great deal of procedural knowledge involved in effective instruction. This 
» owiedge comprises the basis for the tutoring strategies module of the ICAI 
,item. Pedagogical knowledge of curriculum sequencing is important, for 
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example in selecting an instructive problem, tailor-made to help students see a 
point for himself. Knowledge concerning how to conduct a Socratic dialog may 
also form part of the knowledge data base of the tutoring module. The Socratic 
dialog is a very refined version of discourse. Hence, modern cognitive theories of 
discourse comprehension and production have much to contribute to the 
development of tutoring systems. For example* 

Question a prediction made without enough information. 

If: a student makes a prediction as to the value of the 
dependent variable on the basis of some set of factors, 
and there is another value consistent with that set of 
factors, 

Then: ask the student why not the other value. (Collins, 
1978, p.3). 

Other ICAI issues. Other issues of concern to intelligent CAI besides the 
three basic ICAI modules include (a) student-system interface issues, e.g., the use 
of natural language and all of the problems associated with generating and parsing 
prose text, (b) self-improving tutors, e.g., ICAI systems that learn how to improve 
/ their instructional effectiveness, based on experience, and (c) the use of graphics 

as a medium of communication. The natural language problem is a major area of 
M research in its own right, and it is certainly not solved. Means exist to provide 
for natural language interaction between student and system, but they are limited 
and must be treated carefully. Again, there is a resource tradeoff between 
emphasis on this and other areas of importance in ICAI systems. Very little R&D 
on self-improving systems has been crnducted. The achievement of adequate 
baseline performance is still a demanding challenge. (See Sleeman & Brown, 
1982). The presentation to the student of imagery in instructional systems seems 
to be simply a matter of broadening the communications channel between system 
and user and offers no fundamental technical challenges. Having the computer 
"see" images drawn by the student is, however, not yet possible. 

ICAI systems to date have been "hand crafted" and have usually focused 
on less than the totality of issues, concerns, and components of an ICAI system. 
This suggests that practical AI applications must not try to press the state of the 
art on all fronts. Practical efforts should focus on discrete, well defined, and well 
understood content areas that are priority training areas. Computer programming 
is a candidate, but its task domain is not yet fully understood. Troubleshooting is 
a generic area that seem.', to hold great promise as a candidate applications area, 
especially becausi it is a priority training need. Much general work on fault 
diagnosis and isolation has been done; the area telescopes nicely from general 
troubleshooting procedures, to 'electronic troubleshooting, to troubleshooting 
specific systems. The ACTS system v'Hopf-Weichel, Purcell, Freedy, <5c Lucaccini, 
1980) and the SOPHIE system (Brown, Burton, 6c de Kleer, 1982) are two 
completed ICAI projects in electronics troubleshooting. Several current DoD 
i --search projects are continuing research in troubleshooting. See Chapter II. 

For ICAI to be of practical use in the training mission of the Air Force, a 
fully developed, robust system is needed. Demonstration projects will have little 
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utility in the production and delivery environment. Systems must be designed to 
be used, not to servn as research vehicles or demonstrate state-of-the-art ICAI 
techniques. This means that, as a part of the design and development of a 
practical ICAI system, the field and school must have considerable input from the 
outset. This approach is being used by the STEAMER project at NPRDC, where 
interim field tryouts are now underway. 

The fact that ICAI systems are currently hand craf ted, and that the first 
few applications environment-oriented systems will also be hand crafted, should 
not deter a program focused on the development of tools that will be useful in 
moving toward a mass production capability for ICAI systems. Recall that the 
design and development process for traditional hard copy and CAI materials is 
still largely an art. The techniques are well specified, as in the I5D process, but 
little or no automated design help is yet available. 

Task analysis. The techniques reviewed above for automated assistance 
to the design and development of traditional materials, especially the central role 
of the task analysis, will probably have a payoff for the design and development of 
ICAI systems as well. In particular, a large effort focused on developing a 

telescoping task hierarchy" for troubleshooting and fault diagnosis tasks might 
serve as the first step toward design assistance in CAI and ICAI as well. By 

telescoping" is meant a representation that is capable of representing: the 
troubleshooting processes involved in a particular system (e.g., Fill 6883 
Avionics Test Station)} which also serves to outline tne troubleshooting process at 
the next level of generality (e.g., avionics test stations); and so on, to the most 
general level of detail (e.g., general troubleshooting strategies, approaches, and 
procedures). 

Building blocks and tools. Specialized building blocks and tools for a 
particular applications area (e.g., troubleshooting or computer programming) may 
be necessary. Natural language interfaces, pattern matchers, plan recognizers, 
production system interpreters, augmented transition network compilers, and 
graphics editors and graphics generators are examples of more general purpose 
building blocks and tools. A carefully thought-through building block development 
project should parallel the first few ICAI systems development projects. Given 
the centrality of task analysis, automatic or semi-automatic means of "knowledge 
extraction" become important, for a task analysis is an explication of a subject 
matter expert's expertise. The AFHRL Interactive Graphics Simulator project has 
demonstrated a technology for semi-automatic knowledge extraction, which is an 
integrated part of an instructional materials design, development, and delivery 
system. Finally, instructional design itself is a target area for knowledge 
extraction, for the procedures inherent in the design of good instruction play an 
important role in the tutor module of ICAI systems. Certain canonical approaches 
to instruction may be identified, and their procedural aspects codified in on-line 
representations. 

Use of graphics. Graphical or pictorial imagery (objects, structure, or 
process) are a major means of communication. Imagery is so important in mental 
processing that there is great debate in cognitive psychology as to whether 
encoding in long-term memory is in terms of propositions, images', or perhaps both 
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(Pylyshyn, 1973). However, when a technician's job involves working with actual 
objects and pictorial representations of such objects, graphical or pictorial 
capabilities in instructional ystems are necessary. 

The recent advent of the video disc and recent improvements and 
accessibility of computer generated graphics systems make the incorporation of 
imagery into instructional systems practical. Indeed, it may even be possible to 
replace large assemblages of equipment, in particular electronics equipment, with 
graphics simulations. This has important implications for training. More hands-on 
practice can be provided (albeit simulated) than might other , e be possible with 
expensive physical hardware. Also, since these graphics tis are designed 
from scratch, it is possible to design them for the purpose: . oing, such that 
hour for hour, more learning may go on through interaction *ith a graphics 
training device than with the actual equipment. These issues are now being 
studied in numerous OoD research projects. 

Graphics or imagery CBE systems are functionally identical to CBE 
systems in general. The same distinctions between frame-oriented CAI, drill and 
practice, and simulation apply. The same opportunities for ICAI exist. Graphics 
and imagery simply provide an additional modality of communication. This is an 
incredibly powerful and important modality; therefore, it should be exploited to 
the maximum. Approaches to the development of quality interactive instruction 
in general apply to the development of quality video disc or graphics-based 
instruction in particular. Design and development problems should be attacked at 
the most general level possible, e.g., at the level of "interactive instruction"; 
effort need not be duplicated by addressing fundamentally the same issues at the 
more specific level of detail of "graphics simulations." Therefore, the approaches 
to design, development, and delivery suggested in preceding sections apply to 
highly graphics-oriented systems as well. 



Performance Measurement 



Challenges 

Roles of performance measurement in the military include job 
proficiency assessment, ti aining effectiveness assessment, personnel selection and 
classification, certification, promotion, and research and development. The focus 
of job proficiency assessment may be on the individual, team/crew, unit, service, 
or mili'.ary-wide readiness. The roles or pjrposes of performance measurement 
define different mixes of performance tests and administrators. For example, 
OJT performance measurement may be based on very informal and subjective 
judgments on the part of a shop supervisor, who needs to know whether an airman 
can perform a certain task indicated in an Air Force Job Proficiency Guide. This 
approach differs greatly, for example, from the performance measurement 
techniques anc 1 data employed by the R&D community, where data requirements 
are more rigorous, or needed in norm rather than criterion referenced formats. 
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Career specialty characteristics also influence applicable performance 
measurement techniques and procedures. For example, career specialties with 
critical tasks, or tasks associated with high safety risks or high costs, require 
close specialties monitoring through performance measurement procedure?. Each 
Air Force base now has a quality control group whose duty it is to monitor certain 
critical career specialties* 

Technical career specialties, especially maintenance, require 
performance monitoring because modern weapons systems are technologically 
advanced. These systems are complex and maintenance is often far from 
straightforward, requiring considerable troubleshooting skill. Ineffective 
maintenance practices can cost the Air Force exhorbitant funds in erroneously 
replaced units. Moreover, mission preparedness is vitally linked to adequate 
maintenance capability. The Air Force uses about 28 percent of its workforce and 
between $5 and $7 billion annually on maintenance (Townsend, 1980). 

Excessive maintenarce costs can be reduced if job task performance 
tests or empirically valid symbolic substitutes to them are available to ascertain 
how efficiently and effectively maintenance personnel perform their tasks (Foley, 
1977). Performance measurement is especially feasible in maintenance areas 
because of the technical nature of these tasks. Performance is the result of a 
structured application of knowledge, not nuance or intuition. In other words, it is 
possible to obtain good definitions of the goals and methods of performance in 
these areas. The task analytic approaches discussed previously can be used to 
develop, specif iy, and sample specific, concrete, performance measures. The 
hierarchical nature of task analyses is useful in developing sampling strategies. 

The remainder of this section is focused on individual job proficiency 
assessment in technical career specialties, especially maintenance. Measures of 
individual job proficiency may be used in turn as inputs to training program 
assessment. 

During and after World War II, the Air Force made extensive use of 
formal job task performance tests. For reasons of economy, these tests were 
abandoned in favor of paper-and-pencil theory and job knowledge tests. 
According to Foley (1977), "none of these substitute measures are sufficiently 
valid for use as substitutes for job task proficiency tests." This is based on low 
correlations obtained between job task performance tests, theory tests, job 
knowledge tests, and school marks. 

There are a variety of job proficiency test formats. Some numeric 
scores for performance (for example, number of problems correct) have no 
interpretive value. The only useful presentation of job performance data must be 
some meaningful profile of proficiency. Foley has developed an exemplar of such 
a profile. Performance is broken down into various functional areas: use of test 
equipment, repair, remove/install, adjust/align/calibrate, troubleshoot, and 
checkout. Hierarchical dependencies between these various maintenance 
functions are identified Special test equipments use is seen as a component 
function of adjusting, aligning, and calibrating equipment. The adjust/align/ 
calibrate functions are, in turn, seen as component functions of 
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troubleshooting. In this formalism, troubleshooting b explicitly identified as the 
highest order function. 

Job performance tests require training and on-the-job time. Valid 
symbolic substitutes to formal hands-on job task performance measurement may 
reduce assessment costs. Paper-and-pencil symbolic representations of a task 
have potential for reducing the need for job task performance testing on actual 
equipment. Much development has occurred in the past several years on 
simulation systems for maintenance training (Orlansky & String, 1981). Current 
developments in this field may permit the use of maintenance simulators for 
performance evaluation and certification of maintenance personnel on an 
objective basis in operational environments. 

In sum, the challenges are to move performance measurement into the 
operational environments and to integrate it with personnel management and 
training organizations' programs. The development of symbolic substitutes, based 
on modern maintenance simulation work (such as interactive graphics simulations 
utilizing intelligent video discs), also provides a chalienge. 



Approaches 

The development of valid task analyses which form the basis for 
performance measures can be approached utilizing the methods and techniques 
outlined in the previous section on task analysis. Since troubleshooting 
proficiency is a job capability of interest, task representations appropriate to the 
genuine problem solving type of cognitive skill are appropriate. AI techniques 
could be applied to assist in the abstraction from a number of sample task 
protocols of a generic representation for the task. This is similar to the "plan 
recognition" capability that would be of use in ICAI systems. 

From a completed task analysis, performance test problems can be 
constructed, utilizing the hierarchical nature of the task analysis as a guide in a 
sampling strategy that may focus on criticality, difficulty, or breadth of 
coverage. These problems could be administered on actual equipment, or even 
developed as paper-and-pencil tests. But by far the best medium would be 
interactive graphics simulation so that the data collection will be automatic, 
unbiased, consistent, and immediately incorporatable into an integrated 
performance assessment/training system. By replacing actual equipment with 
imagery under computer control, an individual's performance protocol can be 
automatically integrated into a task representation used to characterize that 
individual's expertise. A problem sequencing component may be used to prescribe 
a sequence of OJT practice problems, delivered as interactive, instructional 
simulations, to remediate specific detected inadequacies in the subject's 
performance. In this way, the next tests that are selected for administration to 
the subject will be derived from a representation of the career specialist's 
technical competencies. 

As mentioned, traditional job proficiency task performance tests do 
require on-the-job time and resouices. The approach suggested mitigates against 
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this problem in two ways. First, special attention for methods of task sampling 
across the job tasks can be developed— automatic methods that act on the online 
job representation. In this way, the amount of performance testing may be 
reduced. Second, since performance measurement and OJT are administered on a 
special purpose device, current demands generated by OJT and assessment 
programs on supervisory time and actual equipment are alleviated. 



Job Performance Aiding 



Challenges 

Material in this section was abstacted from Booher (1978). The Booher 
report is a survey and conceptual analysis of existing job performance aid 
techniques, and identifies and categorizes factors important to the selection, 
design, cost-performance tradeoff, conduct of future R&D, and Implementation of 
performance aiding technology. 

Due to Increasing personnel costs and declining entry-level skills, there 
is a great need to enhance the performance of lesser-trained or lesser-skilled 
personnel. Other R&D projects have found that job performance aiding has major 
potential for increasing job performance and for reducing associated costs. 
Analyses reported in Serendipity (1969) and Braid (1975), Rowan (1973), Shriver and 
Hart (1975), and Shriver (1977) indicate that, through the use of job performance 
aiding technology, first-enlistment training costs may be reduced from 50 to 80 
percent, a significant savings in spares may be achieved due to reductions in 
erroneous component removal, and a 15 percent decrease in the maintenance labor 
fcrce may be effected. These savings total $1,675 million annually. 

Job performance aiding liteature has a long history, and definitions of 
JPA go back as far as 1962. Booher (1978, p. 3) proposes the following definition: 

A job performance aid is any device, manual, guide, or tool 
used cr. the job to facilitate performance or to avoid costs 
where learning is incidental. 

Although an important aspect of JPAs is the fact that learning is incidental and 
not a primary product of their use, it is possible to design aids that encourage 
greater incidental learning. The interrelationship between JPAs arid training is 
treated in the final section of this chapter. 

The types of generic JPAs include troubleshooting, nontroubleshooting, 
c*id various other types: standard operating procedures, periodic action aids, time 
critical or hazardous conditions aids, one-trial learning aids, and aids for highly 
complex integrated systems. In addition to types of JPAs, there are also levels of 
aiding. Troubleshooting aids vary from fuily proceduralized JPAs, to partially 
proceduralized aids, to simple logic deductive aids, to systems descriptions. 
These represent a continuum from prescriptive aids (which tell the technician 
exactly what to do each step along the way depending on the exact circumstances 
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that pertain), to descriptive aids which offer no procedural assistance and which 
are principally the source of help from which precise actions must be deduced, 

3PAs are usually designed with particular personnel qualifications in 
mind. Fully proceduralized aids are more appropriate for less experienced 
personnel; m^re deductive aids, for those more Qualified Hybrid aids are 
designed to be adaptive to the level of expertise ot their users, and support 
performance in less proceduralized formats and more deductive formats as the 
user increases in competence. In summary, JPAs are means of extending human 
capability by providing a means of storing for later retrieval information related 
to on-the-job performance. JPAs may reduce or even, in some circumstances, 
eliminate the requirement for technical training. 

Booher found that satisfactory aiding techniques exist for all 
nontroubleshooting operation and maintenance tasks and for most troubleshooting 
tasks on mechanical, electrical, and electromechanical systems. Howeve , no 
aiding techniques were satisfactory for complex digital electronic systems. 
Troubleshooting complex digital electronic equipment, therefore, has been 
identified as a challenge area for both training and job performance aiding. 

Timely and accurate production, distribution, and maintenance of JPA 
systems have been identified as major weaknesses of JPA systems. The major 
cost factor of JPA systems is front-end analysis and the amount of graphic 
enrichment needed to increase intelligibility. 

It was found that microform and audiovisual systems are inferior or 
equal to hard-copy print media. For most job environments, simple hard-copy 
print n.edia JPAs meet the information aiding needs of operators and technicians. 
Further research is warranted into devices that can increase the performance 
capability of the user beyond that possible with hard-copy print. 

Interactive JPA devices can be considered in a class by themselves. 
Burkett and Kruse (1977) have postulated that use of such aids may cause a 
reorganization in the thinking processes of technicians, enhancing the users' 
capacity to develop rnd organize thoughts, define necessary steps, and develop 
the most efficient procedures for problem solutions. Also, these aids, through 
embedding in their target systems, have the potential for modifying, not simply 
aiding, task performance. Only interactive JPAs are identified as having high 
potential is assisting the performance of cognitive problem solving tasks. 

JPAs have many of the same logistical and quality assurance problems 
does technical training. In particular, without a method for assessing a JPA's 
value relative to other JPAs end other methods of developing operational 
performance capability, objective evaluation of the utility of a JPA system is not 
possible. 

Booher has identified several priorities for future JPA R&D. At the 
systems lev. i, the effects on training, the personnel system, and job design 
variables ' v^uld be studied while holding the given JPA system constant. Work on 
new aic* % device?, basic comparison performance data, and hybrid aids is 
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XTJ52£/? ™ ° f technicaJ ^ truc tions is recommended, because 

uSLfTS? ° f s * llab ' es P er w <*d *>d translating that to * reading grade 

£vel is not an adequate method. Finally, R&D into systems that asVist in 
development and validation of JPAs is needed. 



Approaches 



increased 1 ^ R^L^nhifi, 31 1 e i ectr0 " ics J» • Potential priority area for 
increased jfa R*D emphasis. Interactive JPAs were identified as the onlv 

£%?£8L 15! .^toP^nt of interactive computer-based 3PAs for 

D^S fo.^lf h °° tl,lg 15 jested. Such interactive JPAs have gre£ 

El^rnJ^S# to if? r<nt,y in , vesti 8 a ting the development of a Personal 

fr^ume^s 2i (P k? AM) a «"traet with tSS 

~ In u c * w The P rim ary ob ective of PEAM is to produce performance 

spe„iJcations, hardware, and software for a PEAM device. pwwnnance 

applica ^JSrSrS^ 0f ^ slgn ^velopment systems is an important 
tiXii??2? goal for oPAs and for courseware. Such systems should facilitate 

lii Ut all/ JPA $yStem mUSt * maintainab <* » out-of-date JPA 

Currentiv R a? d ? ) NR ty , * ntel "« ibi l ,it L WCre ldentified as ar *» of need. 
J-urrentJy at ONR, a nurr.ter of related pro ects are underway, for examole* 

^prehension .and I Analysis of Information in Text and PfctorW fcta£tol33S 
Bourne, Kinvich & Kozminsky, J981); Structual Approaches to Procedural LearniS 

tS^SZ^^ 1981,5 ^W"^^ 

lK^mi^ w h ! ; J 0pi ? ^ Th T e ,dentifica tion in Prose Comprehension 
StaSfei!! . * K5** 1 ?? ^P 16 Know About EI «ctronic Devices: A Descriptive 
r«ui y ti of t?A 198 l ) * k Mu S ? thiS work has co * nitive science foundations and the 
JPA « JX .T^" ,** "'^Po^^d into the development of device-based 
prini f'nSdTa 3PAs! devel °P m,snt of desi 8" development aids for hard-copy 

The advent of expert systems has opened a new world of possibility to 
job performance aiding. Expert systems have been designed which have, within 
the r boundary of expertise, human level competence. At the present time 
various manufacturers are attempting to develop intelligent dlagiSSTLSSrt 

autorna Ji?^^ di$t * nctilM * ^tween training, job performance aiding, and 
automated test equipment ar* beginning to biur due to the advent of the expert 
system. This is because the same simulated representations of task and 

wheTher in "TP* *i COre ° f CXpert arC ^^Jm 

whether in training, helping a human do a job, or in doin^ it for him or her. 
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Integrating Training. Measurement. JPAs. and Job Design 



Challenges 

Certain information necessary for job performance is better assigned to 
the "head" via training than to the "book" via HP As (Chalupsky 6c Knopf, I9S7, 
Chenzoff, 1973). Training and operational performance are part of a greater 
whole. As hybrid or adaptive 3PA systems become available, there will be less 
and less reason for separating training and aiding in the initial stages of enhancing 
job knowledge and skills. Teel and Chaney (1965) found that the combination of 
training and aiding produced almost double the performance improvement of 
either method used alone. During the career of a technician, the proper mix of 
training and aiding varies. Integrating training, assessment, and 3PA systems 
recognizes this. 

tradeoffs between 3PAs, training, and job design are often discussed as 
if each of these technologies were equivalent. They are highly interrelated, yet 
no one technology is a direct substitute for the other. For example, while a fixed, 
fully proceduralized 3PA yields cost effective performance for entry-ievei 
technicians, there is a negative cost associated with not training these people. 
This shows up later in long-term career development as an inadequacy in having 
and being able to generalize complex troubleshooting skills. 

3ob performance aiding technology and technical training technology 
share much in common, in particular the need for front-end task analysis and the 
need for formative and summative evaluation. Task analyses designed to cover 
the requirements of all three issues would result in cost savings through reduced 
duplication of effort and in increased effectiveness, through explicit recognition 
of the close relationship between training, job performance aiding, and job design. 



A clear challenge is to develop sophisticated design systems that 
formally take these issues into consideration in the development of residence 
training, on-the-job training, and 3PAs. In addition, when new weapons systems 
are being designed, the induced effects of device and job design on the training 
and aiding requirements can be explored and a human-factored design may be 
approached in a systematic fashion. 



Approaches 

A kernel, procedural knowledge base can be developed for an existing or 
in-development weapons system that includes the device representation, a task 
representation, and a how-it-works representation. This knowledge base can be 
used tojnvestigate tradeoffs between job performance aiding, residence training 
and OJT, and* device and task design. Such analyses will yield empirically based 
decisions regarding device configuration, job design, ct\<i training and performance 
aiding requirement.;. It is then possible to utilize this knowledge base in concert 
with appropriate design systems, building blocks, and tools in the design, 
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development, and delivery of adaptive instruction and hybrid JPAs. As seen in 
Clancey's experiments with MYCIN, it is not true that a knowledge base 
especially suited to performance would be necessarily useful for instruction. But 
that should not be a problem in this case if the knowledge base is designed to 
support a basis for experienced technicians to expand their skills, whether in 
training or on the job. Systems which merely require the Input of symptoms and 
respond with the probable fault are not useful in this regard. The Air Force goal 
is to builo a basis for technicians to improve their performance and to be able to 
transfer their skills from one job position to another, contributing to a long-term 
enhancement of the level of human expertise on the job. Therefore, an Integrated 
adaptive training/ajsessment/JPA design, development, and delivery system 
addresses the ultimate source of need for training and job aiding in the first 
place— a shortage of expertise. 

If this approach Is chosen, It Is criticaJ that a means of reducing the 
labor intenslveness of developing and using the central data base be developed. 
At the very least, tools and building blocks should be developed to facilitate the 
construction and use of such knowledge bases. Approaches to semi-automatic 
knowledge extraction have been mentioned in the section on task analysis. It is 
critical that the central data base be able to accommodate the variet" of views 
and perspective? and have appropriate interfaces with each user community, so 
that each can effectively interact with the data base-in defining it, building it, 
refining It, validating it, and using it. Members of the various user communities 
must be able to interact with the data base individually, and directly, otherwise 
its maintainability Is compromised. A system that cannot be supported will not be 
used. And a system that is designed to be used by many people with different yet 
closely related goals will not be supported unless these people have direct access 
to it. Direct user involvement contributes to a feeling of "ownership" that goes a 
long way toward assuring that the system will be accepted. 

With a community involvement with a common data base, strong, viable, 
sjrstemic mechanisms must be in place to ensure that the data base and its 
products (training, JPAs) are valid and remain valid. In addition to traditional 
evaluation procedures, the central knowledge base can be used to develop 
measures of perfor nance, and the various delivery systems can be used to collect 
performance data, with which to validate the tradeoffs chosen and the utility and 
effectiveness of the training and 3PA system which have been designed. 

This integrated system can be extended to incorporate 03T management 
procedures and the tracking of individual personnel qualifications at a level of 
detail useful in maintaining an ongoing OJT program for e particular technician. 
Data on individual qualifications maintained in this integrated system can be 
interfaced with related management systems, such as the Integrated Training 
System (ITS) and the Training Decisions System (TDS). These interfaces can be 
developed utilizing A I concepts and tool*, which may contribute to the 
management of complex systems. 
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CHAPTER IV. PRACTICAL CONSIDERATIONS 



Early In its history, AI hHd forth great promise of near-term 
applications, for example, in language translation and chess. The failure of this 
promise to materialize was a great blow to the credibility of AI research. In the 
interim, as discussed in Chapter I, AI has matured, particularly through emphasis 
on knowledge-based systems. Now, once again, AI is holding forth promise. 
Numerous private, for-profit ventures seeking to capitalize on AI applications 
have recently been formed. Public awareness of AI lias been heightened by the 
Japanese "fifth generation computer" project. DoD has taken a greats interest 
in near-term applications, an interest which this report serves. It follows that AI 
procurement commitments must be carefully targeted for success. A second 
failure for AI on its second rising would be a tremendous blow to AI and to these 
who sought to apply it. 

The remainder of this chapter is organized around the following practical 
issues: available hardware, software tools, personnel resources, and guidelines for 
practical efforts. 



Hardware 



Until the last few years, almost all work on systems of the sort described 
here has been performed on computers whose typical million dollar price tag h . 
been a barrier to both R&D and practical use, Many sites have tried to use 
existin" data processing facilities, but these systems did not offer suitable 
languages or interactive capabilities and were ruled out immediately. 
Fortunately, a selection of much less expensive equipment now exists in price 
ranges acceptable at least for the development phase, and trends indicate that 
target machines for delivering this type of system at low cost are just around the 
corner. For example, for development purposes, high performance personal LISP 
machines (such as the LM-2 and 3600 systems available from Symbolics) or similar 
machines could be used. Other comparable systems include the Xerox "Dolphin" 
and "Dorado," and the LMI CADR. r he Xerox machines support INTERLISP and 
arc compatible with the "Dandelion," a possible delivery vehicle. The Three 
Rivers PERQ has many similarities to the systems just mentioned, but currently 
has no LISP implementation. 

It is very likely that M680O-based systems will begin to be available with 
production-quality LISP environments, once the 68010 (virtual memory) version of 
this micro-processor begins to be widely available. The following systems could 
serve as lew cost target systems at that time: Apollo Domain, Sun Workstation, 
WICAT 150, Fortune, Corvus Concept, and many others. Few have virtual 
memory at present. (Large virtual memory is the primary architectural 
requirement for A I.) In seme specialized situations, the IBM Personal Computer 
might be usable as an AI target machine if equipped with Winchester Disk and 
sufficient memory. These systems at present have no production-quality LISP. (A 
number of LISP implementations, including one at Utah and one at Yale, are 
underway at present.) 
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Software Tools 

Creating an AI application means, among other things, creatine 
^;Irm« e, K^i° ftWare languages, environments, and higher order development 
^i?^ti^ nr 5L, CrUCial . wnilde ™t»°n« ^ the practical development of an AI 
application. This section considers, in turn, whether LISP is essentially a 
EfSvlI? Ration for AI programming and what higher order development 
systems are available and, ultimately, how useful these can be. 

Despite the benefits of widely available (and allegedly more efficient) 
anguages such as FORTRAN or PASCAL, or the structure and porSu^^&L 
the current technology of AI employs the use of the LISP language. Automatic 
storage management, the elevation of functions and closures to "first class 
Sir*" i t ? tus ' the incept of property lists, the relationship between variables 
CLZ^Sfc ™? ° entral . r0lC ° f m lnter Pret«r (rather than a compiler) are 
features of the language arguing for its use in this type of application. Other AI 
languages have been developed (Barr & Feigenbaum, 1982), but LISP in its various 
dialects predominates as the base language. 

a.n. ra i • A ! t ^ b 1 t u eS . t !l at , characterize Al-supportive programming languages in 
general include the following: 

1. Focus on symbol manipulation and list processing. 

2. Support of representations, which change dynamically. 

3. Support of flexible control by pattern matching rather 
than by procedure calls. 

4. Supportive programming environment, including 

a. An interactive (interpreted) language. 

b. A good editor (program construct oriented, not text 
oriented). 

c Debugging facilities (traces, breaks), 
d. Standard systems input/ouiput functions. 

High Order Development Systems 

There are numerous A I "building block" systems available, some from 
universities without support, and some commercially. An expert systems 
workshop (funded by the National Science Foundation and the Defense Advanced 
Research Projects Agency) was recently held, resulting in two important products. 
One (Waterman & Hayes-Roth, 1982) is a report describing an investigation into 
the comparative merits of eight very high level programming languages designed 
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for expert systems: EMYCIN, KAS, EXPERT, OPS5, ROSIE, RLL, HEARSAY-III, 
and AGE. 



The other product is a book describing how to build expert systems 
(Hayes-Roth, 1983). This comparative study describes available systems, their 
sources, their comparative merits, and lessons learned from a comparative 
analysis of these systems, in particular, insights regarding choosing a tool 
appropriate for the problem and designing a tool for expert system building. 

Quoting the comparative study (Waterman & Hayes-Roth, 1982, p. 07), 
maxims suggested for choosing an appropriate tool are: 

1. Do not pick a tool with more generality than you need. 

2. Test your tool early by building a small prototype 
system. 

3. Choose a tool that is maintained by the developer. 

*. Use the problem characteristics to determine the tool 
features needed. 

Various tools exist for knowledge representation, in particular, KRL 
(Bobrow & Winograd, 1977) and SRL (Wright & Fox, 1982). 

For specific applications, one should consider developing tools and 
building blocks from scratch. An expert, using a LISP machine, could develop an 
inference/retrieval package in a week or two of intense effort. The resulting 
package is apt to be more suitable for the specific application than might be the 
general packages now available. However, in the long-term, the i«se of such 
building blocks as components in larger systems will be essential. 



Personnel 



People with formal training or practical experience with AI are 
extremely rare. The recent interest in AI by the private sector has created an 
enormous demand for AI experts, creating, for the experts, a sellers' market. This 
means self-aware AI experts expect high salaries, interesting work assignments, 
software and hardware environments they are familiar with and like to use, and 
evidence of management commitment to their work. Because of this, other 
practical considerations, such as choice of hardware or software are often forced 
by personnel choices. This is especially germane to the LISP issue: does one need 
to do AI in LISP? The question is moot if it is true that most AI experts like to 
and are accustomed to programming in LISP and may not be interested in working 
on a projejt that does not use LISP or one of the higher level tools built upon 

The common advice from people involved in AI projects is to recruit 
people with proven AI expertise and not simply to declare r.omeone in the c'ata 
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E2!? ng (tepartm ? nt f* ihe AI expert. At the same time, due to the shortage of 
,*? ^Kanlzation should make every possible effort to develop its own 

SVTA W , SUgge$t3 "» AI e,fort * ^ a? ollows: 

SSihl^Sf £ liL the . main initial technology requirements, use the best of 

wUl rirJtf t^h„^'n 0l0g ^^ PP ? rt - Wi l h this strate 8y' the follo * in « dynamics 
rJ™£? m te f hni « al, y t^ted but P robab| y managerially unsophisticated AI 
recruits will mature in leadership qualities as the existing staif , ^ready attuned 
to organizational realities, acquis technical competence in AI. 

The personnel /crunch" is widely recognized by managers of DoD 
ffin. !!S A solution to the shortage is addressed most directly in a 

training and education program proposed to the Joint Logistics Commanders. This 

^T^XiS^V^ £ inCreased hi * her Nation opportunities inAI 
and hopefully help to alleviate the undersuppiy of trained personnel in this field. 

A * r "f ntioned ' Al 15 becoming a commercial venture; so, to « certain 
R^^HFll? be . direC ? ly extracted for with firms active in this area. 
Reports on those firms have been made in the popular press (cover story in 

fn"^ ^' ^ rCh ,\ I982; a ma ' or "■I 161 * in .lS& May 17, 1982)! 01 £Zs* 
n^iSf* 1° ^ ° f competence, there are the basic university 

nonprofit and governmental centers of activity. These too may serve as sourcei 
oi expertise and potential sources for recruitment. 

Guidelines for Practical Efforts 

* a „ Kt T ,° day ' S a PP lications Promise of AI is real, but past overzealousness has 
taught a lesson. In any applications effort contemplated today, the following 
principles of practical and achievable project management must be followed. 

. . „ T, hC , SC *° pe 0f the a PP iicat ions effort must be constrained to highly 

specialized solutions to narrowly defined problems. From Hayes-Roth (Gevarter, 
1>8Z, p. 2)1 the problem should be nontriviai but tractable, with promising 
SIS?! fw » IM 2 emen ? 1 ^pansion." Objective, definable interim milestones 
should be set. Hayes-Roth (1982, p. OiCMii) lists some situations that instigate 
knowledge engineering initiatives and some kinds of product innovations now 
possible: 

Situations that Instigate Knowledge Engineering Initiatives 

1. The organization requires too many skilled people to 
recruit or retain, 

2. Problems arise that require almost innumerable 
possibilities to be considered. 

3. lob excellence requires a scope of knowledge exceeding 
reasonable demands on human training and continuing 
education. ° 
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Problem solving requires several people because no 
single person has the needed expertise. 



5. The company's inability to apply its existing knowledge 
effectively now causes management to work around 
basic problems. 

Kinds of Product Innovations Now Possible 

1. Active instruments that solve problems by producing 
answers instead of data. 

2. Systems that explain how they work and how to use 
them. 

3. Corporate memories. 

4. Reasoning aids and thinking prosthetics. 

5. Accident-proof machines. 

6. Automated sellers. 



7. Hypothesis and expectation management systems. 

Realistic project cost and personnel effort estimates are crucial inputs 
to successful applications. These costs depend on the availably of tools. 
Hayes-Roth (1982, p. 412) cites as the resources required for a typical knowledge 
engineering project (about 500 knowledge "chunks" in the rule base), utilizing 
existing tools, to be about $300,000 to $1,500,000 with 2 to 5 personnel years. 
Included in this estimate is design, development, knowledge engineers, computing 
and overhead. 



Another analysis, based on project costs of exemplary AI projects over 
the past 15 years led Davis (1982, p. 10) to the following cone usion: 

Even for the best-understood problems, experienced 
researchers using the best-understood technologies still 
require at least five man-years to develop a system that 
begins to be robust. 

The stages of development of an expert system include (Davis, 1982) 
system design, system development (conference paper level), formal evaluation of 
performance, formal evaluation of acceptance, extended use in prototype 
environment, development of maintenance plans, and system release. Davis notes 
that, to date, no current system has completed all these stages. 



Case Study: Practical Guidelines for AI Applica*ions Efforts 

In order to add substance to the observations regarding development 
guidelines, this paper includes as a case study the considerations paid those issues 
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by one of the first commercial attempts to capitalize on AI technology, 
Computer* Thought Corporation's Ada Tutor. 

• 

This section describes the current Computer* Thought product 
development effort as a case study In practical AI-CAI applications projects. The 
reader should be aware that the project is not yet complete, and so conclusions 
must be treated as tentative and subject to modification. The material in the 
remainder of this chapter was provided by Dr. Mark L. Miller of 
Computer* Thought. 

How was the scope of the practical applications effort determined and 
specified? 

The founders of Computer* Thought had been involved, over a number of 
years, in developing and working with prototype systems for teaching computer 
programming. It was desired to apply this technology to some need that would be 
sufficient to warrant the cost of using systems in the Vax or LISP machine class 
for initial deliveries, on the assumption that personal computers with this 
performance will be commonplace in the mid to late eighties. With DoD software 
expenditures in the billions of dollars per year, and many embedded systems 
contracts involving millions of dollars, the Ada training market provided an ideal 
opportunity. Computer*Thought is developing a generic AI-CAI technology for 
teaching computer programming, using Ada as the initial vehicle. As hardware 
costs continue to decline, programming tutors for new, potentially mass-market 
languages (e.g., LOGO or SMALLTALK) wUl be natural follow-ons from this 
technology). The project scope (In terms of both curriculum and system features) 
was narrowed until a small team (growing to a dozen technical staff) could 
complete an initial product implementation within 1 year. 

What effect did the following conditions have on the effort? 

1. The specificity of the problem. 

Probtem specificity is the key ingredient to success in building this 
type of system at the present time. As the project has progressed, 
the system designers have continued to revise initial requirements 
to increase specificity. There are mbJiy topics one might include in 
an Ada curriculum, for example, and many features one might 
include in such a system. The question, "will this have a first-order 
effect on overall effectiveness?" has helped to focus system goals. 

2. Existence of prior related R&D and prototypes. 

The Ada*Tutor project was specifically chosen because the company 
founders had experience with several such Ricu prototypes. In 
particular, the BASIC Instructional Program (BIP), the Spade 
system, ACSES, and TURTLE were previous prototypes which guided 
t.te design. Also, existing experimental literature on help systems 
for learning programming was useful. 



46 



BEST COPY AVAILABLE 



3. Project resource expenditures as a function of time or phase. 

Computer*Thought's initial capitalization has been almost entirely 
devoted to the Ada* Tutor project. Staffing has grown from the 2 
founders to 11 at present and is planned to grow to about 20 by 
summer of 1983. Approximately 50 percent of staff members have 
primarily technical functions. 

Personnel qualifications, background, mix. 

Unfortunately, such projects currently require a rare type of talent 
that is in great demand in industry. Very few universities offer any 
formal training In this area. A team of k to 12 technical people, 
organized around a single technical leader, seems most effective. A 
certain amount of administrative assistance, as well as junior staff 
(e.g., AI graduate students) can multiply the productivity of key 
personnel considerably. It takes several years of intense study to 
achieve expertise in AI/LISP programming; hence reassigning 
conventional computer science personnel to this task, without 
formally trained outside expertise, is not recommended. 

3. Hardware/software tools. 

The level of ambition of the current Ada* Tutor could not have teen 
achieved within the available resources and time were i* not for 
high performance personal LISP machines and th^ programming 
environment they provide. Graphics features such as "windows," and 
interaction features such as menu selection using a "mouse," have 
made it possible to rapidly prototype, demonstrate, and re-evaluate 
ideas within a few weeks. AI programming tools, such as the 
ZMACS LISP-fpecific screen editor and the ability to rapidly locate 
callers/users of a symbol within a large system, have been essential 
fact :r$ in project progress. 

What are the stages in such a project? 

AI product development projects are really too new to reach any sort of 
generalization at present. It seems clear that traditional software project life- 
cycle models do not carry over well into the AI and AI-CAI arenas. Cun'ent 
experience seems to be settling on four more or less distinct phases: (a) 
requirement*; analysis and initial design, (b) implementation of an initial 
feasibility demonstration, (c) prototype development and initial testing, and (d) 
beta testing and final revision. There remains a danger that the level of 
robustness required for the final product (versus a prototype) w'U be more 
difficult to achieve than was originally contemplated. 



47 

BEST COPY AVAILABLE 



CHAPTER V. CONCLUSIONS AND RECOMMENDATIONS 



Conclusions 

Training, performance measurement, and job performance aiding are 
neither separate nor independent Air Force activities. Training (both in-residence 
and on-the-job) and job performance aiding are related because they are 
alternative, complementary approaches to the problem of supporting performance 
capability In career specialties. Performance measurement is related to both 
training and job performance aiding becaus* it is through performance 
measurement that training and 3PAs are validated, evaluated and improved The 
close relationship among these three disciplines is not limited to their 
interdependence. They are also related because scientific approaches to each of 
these three disciplines :.,e based on task analysis. 

« The technology and techniques of AI are based on development of 
specific knowledge representations and sets of procedures that, act on those 
representations. The performance capability of intelligent systems is based on 
the representation and utilization of large amounts of knowledge. 3ob task 
analysis serves* as the means of providing a representation that can, in turn, be 
processed by training, performance measurement, and JPA systems. Therefore, 
the fundamental relatedness of this knowledge may be capitalized on in the 
development of intelligent systems to serve these disciplines. 

From a review of the challenges facing each of these disciplines 
separately, it appears that the development of adequate job task analyses is a 
basic problem. From a review of AI technology, it appears that AI provides a 
unique opportunity to develop and validate representations that can serve as task 
analyses. These representations are machine-readable and can be used with the 
h Si P i 0f AI techni( l ues t0 simulate intelligent behavior, for example: to perform 
♦he job task of interest, or to develop or provide instruction in the job task of 
interest. AI (a) provides a mechanism for developing valid task analyses that help 
solve one of the major problems facing all three of the potential application areas 
of interest, (b) provides a basis for integrating the three applications areas into a 
formal integrated system, and (c) does so computationally, providing the 
foundation for an automated, computer-based system to serve as an aid in 
increasing productivity and effectiveness in training, performance measurement, 
and job performance aiding. 

The principal recommendation to be derived from these observations is 
that training, performance measurement, and jr>S performance aiding should be 
treated as an integrated system, and that an integrated system is the natural way 
to approach AI applications in each of these areas. Since AI applications are 
computer-based; a computer-based system is the required medium in which to 
meet the challenges. 

No Integrated computer-based system for training, measurement., and job 
performance aiding exists. There is, therefore, a great opportunity here to design 
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and build a computer-based system, capitalizing on scientific approaches to 
building a common foundation of task analysis. Whether such a system should be 
non-AI-based or Al-based may be questioned. However, it is relevant to note 
that AI itself is founded on formal representations and the procedures acting upon 
them, and therefore offers a set of tools and techniques that are almost an exact 
match for the requirements of integrating the three applications areas. 

Traditional approaches in each of these areas ha^e weaknesses in the 
area of knowledge representation. AI offers approaches tc neeting the challenges 
faced by each discipline separately and offers a way of integrating these 
disciplines in a formal, explicit fashion. 



Recommendations 



Specific recommendations for the development of an R&D program in 
applications of AI technology to training, performance assessment, and job 
performance aiding are presented below. The approach taken is to structure such 
a program around an integrated technical training, performance measurement, 
and job performance aiding design, development, delivery, and evaluation system. 
The details of specific approaches within each of the areas of training, 
measurement, and job performance aiding are presented in the Approaches 
sections in the main body of this re;>ort. 



Recommendation 1 

A single, integrated, intelligent, computer-based system should be 
developed having application to three facets: training, performance 
measurement, jn<-; pb performance aiding. 



Recommendation 2 

The same system should support in an integrated fashion the design, 
development, delivery, and evaluation of each facet. 



Recommendat 3 

The integrated computer-based system should support and encourage 
direct interaction with instructional developers, subject matter experts, course 
personnel, on-the-job supervisory personnel, evaluators, and the trainee/specialist. 



Recommendation 

A master plan specifying the basic functions of the computer-based 
system should be developed and used to prioritize its own implementation so that 
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the task analysis representations are developed first and that each project effort 
can be integrated with the iarger system. 

Recommendation 3 

As early as possible, the system should be configured as a prosram 
demonstration. r * 



Recommendation 6 

A target career speciaJty should be selected through careful front-end 
analysis with the objective of identifying a job area, first, that is of critical 
importance in the Air Force mission, second, for which there is promise or need of 
improved productivity and effectiveness or of reduced cost of operations, and 
third, that is amenable to the detailed analyses required to develop device and 
task performance models. Avionics maintenance is identified as a general area 
meeting the above criteria. 



Recommendation 7 

The central procedural and declarative knowledge bases should include 
la) a procedural representation of the devices) from the perspectives of both 
operations and maintenance, (b) a procedural representation of the tasks, both 
routine operations and maintenance, (c) a suitably detailed "how-it-works" 
hierarchy regarding the device's function illty and structure, (d) an explicit model 
of the interrelationships between training, measurement, and job performance 
support and evaluation for the career area chosen, and (e) an explicit body of 
procedural knowledge relating to curriculum sequencing and pedagogical 
interaction. 



Recommendation 8 

The system should oe developed in LISP or alternatively, in Ada 
(assuming that comparability can be established between LISP and Ada). All 
aspects of the system should share the same computing environment} hence, tlv 
delivery modality of the system should also be in LISP. 



Recommendation 9 

The system should utilize start-of-the-art graphics, Including high 
resolution graphics and graphic overlays on video disc images. 
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Recommendation 10 

The system'i terminals or stations should be networked. Local 
processing should be supported, but it is necessary for data, in particular the 
procedural and declarative knowledge bases, to be stored centrally. These 
knowledge bases will be undergoing continual development and refinement 
throughout the design, developnent, delivery and evaluation process. 

Recommendation 11 

System hardware should be state-of-the-art LISP machines or carefully 
chosen alternatives. The long-term Implementation strategy should rely on the 
fact that delivery versions of these machines will be on the market in 3 to 5 
years. 
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CURRENT RELEVANT DoD RESEARCH PROJECTS-DETAILED LISTING 

The information in this section is divided into the following fields: 
Project Title 

sponsoring agei*y performing organization project duration 

project monitor project contact level of funding 

telephone number of monitor telephone number of contact 

Reference to material that will provide additional information 



For each title, the maximum amount of information available has been 
included. The information is presented alphabetically by title; Table I may be one 
useful index for locating project titles in Appendix A, as it is arranged by 
sponsoring agency. 

Abbreviations of the sporcoring agencies are as follows: 



AFHRL Air Force Human Resources Laboratory 

AFOSR Air Force Office of Scientific Research 

ARI Army Research Institute 

NAVAIRENGCEN Naval Air Engineering Center 

NPRDC Navy Personnel Research and Development Center 

NTEC Naval Training Equipment Center 

ONR Office of Naval Research 

TAEG Training Analysis Evaluation Group (Navy) 
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Adaptive Computerized Training System (ACTS) 

ARI Perceptronics, Inc. 

B.W. Knerr A. Freedy $521,000 

(202) 274-8872 (213) 884-7470 

nSf^fo!?*" 10 * 1 Information Center. Work Unit Summaries. Alexandria, VA: 

A^yooed Computer-Baaed Training for Propulsion and Problem Solving! 

^5°., , Bolt B<!ranek & Newman, Inc. March 81-March 84 

J.D. Hollan, Jr. A. Stevens $2,198,543 

(714)225-7121 (617) 492-3656 

r^?£ n f Q ]? ChnlCal Information Center. Work Unit Summarie s. Alexandria, VAj 
DTIGj 1982* 

Advanced Instructional Development* Intelligent Maintenance System 

AR J , Sep 83-3an 86 

3. Psotka 

(202) 274-5540 



Analysis of Acquiring Strategic Knowledge in Problem Solving 

ONR (667-459) University of Pittsburgh 

3.G. Greeno & M. Riley 
(412) 624-4141 

Office of Naval Research. Psychological Sciences Division, 1981 Progr ams. 
Arlington, VA: Office of Naval Research, December 1981. 

Artificai Intelligence Analyse*. 

NTEC University of Central Florida Jun 81-Sept 82 

Ricard D. Abbott $201,992 

(305) 646-5130 (305) 275-2216 

Detense Technical Information Center. Work Unit Summ aries. Alexandria, VA: 
DTIC, 1982. — ' 
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Artificial Intelligence-Based Maintenance Tutor 



ARI 

3, ?sotka 



(project in preparation) 



(202) 27003*0 

Artificial Intelligence in Maintenance Diagnostics and Information iystems 



Defense Technical Information Center. Work Unit Summaries. Alexandria, VA* 
DTIC, 1982. 



Automated Performance Measurement System C-3 Aircraft 

AFHRL (Williams AFB) 
R. Hughes 
(602) 988 -6361 

Air Force Human Resources Laboratory. AFHRL An nual Report: FY 81. Brooks 
AFB, TX: AFHRL, 1982. 



Automatic Knowledge Acquisition Techniques for Expert Systems 

NTEC Percept ronics, Inc. 

R. Ahlers (213)880-7070, $80,000 

(305) 606-5130 

Defense Technical Information Center. Work Un it Sum maries. Alexandria. VA: 
DTIC, 1982. 



CA5DAT Enhancement 

NTEC Rowland and Company Sept 81 -Jul 82 

R- B ird N. Marcue $93,831 

(305) 606-0701 (305) 890-7086 



AF'iRL 



Systems Exploration, Inc. 



Ad thorlng Instructional Materials (AIM) 



NPRDC NPRDC 
3.D. Ford W.H. Wulfeck 

(710) 223-7121 (710) 223-7121 



Oct 81 -Sept 86 
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Clow *A W Etotooak Equipmisnt &&»am«iMmc«i Training System 

NPRDC 
V.M. Mafec 
(71*) 223-7121 

Navy PersonncA Research arid Development Center. NPRD C FY-8 2 Programs. 
San Diego, CA: NPRDC, rU. ~ B 

Cognitive and Instructional Pactum In Stos Ao^Mlsitlon and Mnlntenmce of Skills 

ONR (157-430) University of Pittsburgh .Ian 79-Sept 82 

M ' J » F<UT R. Glaser, A. Lesgold, $1,298,950 

xrw , „ 3. Grceno, M. Chi, k M. Resnick 

(202) 696-4504 (412) 624-4895 

Defense Technical Information Center. Work Unit Summaries. Alexandria, VA: 
DliC, 1982. — 

Cognitive Processes in Planning ami Control 

Of 1(667-111) Rand Corporation 

B« Hayes<>Roth 

Office of Naval Research. fsy^logicaLSclenccs Di vision, 1981 Programs . 
Arlington, VA: Office of Naval Research, December 1981. 

Comprehension and Analysis of Information in Text and Pictorial Materials 

ONR (667-422) University of Colorado 

L.E. bourne, Jr., W. Kintsch, & 
P. Baggett 
(303) 492-0111 

Office of Naval Research. Psyc hological Sciences Division, 1981 P rograms. 
Arlington, VA: Office of Naval Research, December 1981. 

Computational Models of Problem-Solving Skill Acquisition 

ONR (667-477) Xerox Palo Alto Research Jan 82-Dec 82 

Center 

H - Half! J.s. Br.wn $98,730 

(202) 696-4327 (415,494-4341 

Defense Technical Information Center. Work Unit Summaries. Alexandria, VA: 
DTIC, 1982. ' 
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Cjmputer Aided Authoring System 

TA£ G University of Central Florida 

Hosni $39,000 
(305) 275-9101 

Computer-Based Maintenance Aids for Technicians 

AFKRL (Wrig> .-Patterson AFB) 
D.L. Thomas 
(513) 255-3771 

Air Force Human Resources Laboratory. AFHRL Annual R eport: FY 81. Brooke 
AFB, TX: AFHRL, 1982. 

Computer-Based Modeling and Diagnostic Testing of Problem-Solving Skills 

ONR (150-508) Carnegie-Mellon University Nov 82 -Nov 80 

H. Halff P. Langley $203,025 

(202) 696-0327 (012) 578-3598 

Office of Naval Research. Psychological Sciences Division. 1982 Programs . 
Arlington, VA: Office of Navai Research, December 1982. 

Computer-Based Program Consultant for Introductory Programming 

ONR/ARI Yale University 3un 83-Sep 80 

R.A. Wisher E. Soloway 

(202) 270-5779 

Personnei Training and Research Group of the Office of Naval Research. News <5c 
Notes, 1982. 2(0). 2-5. 

Computer-Based Techniques for Training Procedural Skills 

NPRDC 
E.L. Hutchins 
(710) 225-7121 

Navy Personnel Research and Development Center. NPROC FY4.2 Proprams. 
San Diego, CA: NPRDC, n.d. "~ 
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Computer-Based Training of Mission Planning and Control Skills 

S^f (1^ S6) Perceptronlcs, Inc. Feb 82-Feb ', 5 

H.M. Halff p.w. Thorndyke $225,000 

(202) 6964327 (213) 880-7070 ^ ^ 

Defense Technical Information Center. Work Unit Summaries. Alexandria, VA: 
DTIC, 1982. 

Computer-Based Tutoring for Diagnostic Skill Acquisition 

ONR/ARI Stanford University Mar 79-Sep 85 

Hv* B. Buchanan & W. Clancey $1,110, W 

(202) 270-55*0 ($15) 097-1997 

Personnel and Training Research Group of the Office of Naval Research. News <5c 
Notes. 1982, 2(0), 2-5. — — ~ 

Development of Draft Military Specifications for Maintenance Task Analysis and 
Logic Tree Troubleshooting Aids 

AFHRL (Wright-Patterson AFB) 
E.G. McFall 
(513) 255-5910 

Air Force Human Resources Laboratory. AFRHL Annual Report: FY 81. Brooks 
AFB, TX: AFHRL, 1982. " 

Evaluation of Individual Performance in Mech&jidcal Specialties 

AFHRL (Brooks AFB) 
S. Lipscomb 
(512) 536-3551 

Air Force Human Resources Laboratory. AFRHL Annual Report: FY 81, Brooks 
AFB, TX: AFHRL, 1982. 

General Patterns of Scientific Inference 

ONR/ARI Carnegie-Mellon University 

3. Psctka J. Larkir. (c 3. Carbonell 

(202) 270-5500 (012) 578-2000 

Personnel and Training Research Group of the Office of Naval Research. New? A 
Notes, 1932, 2(0), 2- ' ' 
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Human Understanding of Complex Systems 

ONR (667-428) Bolt Beranek & Newman, Inc. Mar 79-Feb 82 

/%& JSJ , A,M - Collins * A - Stevens $292,531 

(202) 696-430* (617)097-3377 

^IC^982 ChnlCal Information Center " Work Unit Summaries . Alexandria, VA: 
Jrrtell^ent^irtomated Tutors for Instiuction in Planning and Computer 

2 m R ulut 6i) I, exas Irtstrumen ". Inc. (exp date) Sept 81 

JSK Hal,f M. Mliler $99,500 

(202) 696-4327 ( 2 10) 980-6081 ' 

DTIC^Jr^ 1 ^ 1 UdormAtion Center * Work Unit Summary . Alexandria, VA: 

Intelligent Computer-Assisted Instruction: An Interactive Reading Tutor 

21* Yale Univer »"y 3un 83-Sep 80 

R.A. Wisher R. Schan.t & J. Black 

(202) 270-5779 (203)036-^771 

SKt? R «earch Group of the Office of Naval Research. News & 

brteUisent 0>mputer-flased Instruction for Simulation Training of Troubleshooting 

? N a R {^ R ,! Bolt ^ranek 3c Newman May 83-Sep 80 

K^ox Wi ? ner , W. Fuerzeig «5c 3. Frederiksen 

(202) 270-5779 (617) 097-3008 

No^m2?2(0) a 2 n -5 n8 ReSearch G ° UP 0f the ° f fice of Nava! Research. N ews & 

Intelligent Instructional Methods for Teaching Procedural Skills 

ONR (150-093) Bolt Beranek 4c Newman, Inc. 

W. Fuerzeig <Sc 1. Frederiksen 
(617) 097-3008 

Office of Naval Research. Psychological Sciences Division, 1982 P rograms. 
Arlington, VA: Office of Naval Research, December 1982. — ~ 
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Intelligent Tools for CAI 

ARI Yale University 

Soloway $30,000 

Interactive Computer Grahpict Simulation for Intermediate Level Maintenance 
Trainer 

D AFHRL (Lowry AFB) Canyon Research Group, Inc. May81-Sep84 

j a. Deignan B. Pieper $700,000 
(303) 370-3391 (303) 753-3376 

, « 

g Memory-Based Expert Systems 

'.! AFOSR Yale University 3an82-Mar84 

W.R. Price R. Schank $139,178 
(202) 767-5023 (203) 436-4771 

•j Defense Technical Information Center. Work Unit bum maries. Alexandria, VAj 
$ DTIC, 1982. 
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Mental Representation of Circuit Diagrams 

ONR (66 7 -491) Perceptronics, Inc. 

M.G. Samet & C. Wickens 
(213) 884-7^70 

Office of Naval Research. Psychological Sciences Division. 1981 Programs . 
Arlington, VA: Office of Naval Research, December 1981. 

A Model for Procedural Learning 



i ONR (154-399) Carnegie- Mellon University Sept 78-Sept 80 

M.J. Farr 3.R. Anderson $174,987 

• (202) 696-4504 (412) 578 -2781 



Defense Technical Information Center. Work Unit S ummaries. Alexandria, ,( A: 
DTIC, 1982. 
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Modeling Student Acquisition of Problem-Solvif^ Skill* 

2^ R ( J Rut 8 ert University Sept 79-Aug 8i 

H.M. Halff , R.L Smith $129,992 

(202)696-0327 (201) 932-3623 

Rr f |? S fol5 <JwiCal Information Center. Work Unit Summaries. Alexandria, VA: 

Personal Electronic Aid for Maintenance (PEAM) 

^ T ^ c Texas Instruments, Inc. 

B. Rlzzo $100,000 

(305) 606-5130 *iuu,uvu 

Plan lor Application of Artificial Intelligence Technology to Technical Training, 
Performance Measurement, and Job Performance Aiding 

£ FHRL Denver Resea-ch Institute 3 an 83 -Sep 83 

G.Walker 3. Richardson $125,000 

(303) 370-3391 (303) 753-2087 

Principles of Intelligent Computer-Eased Instruction for Cognitive Skills 

ONR/ARI University of Pittsburgh Mar 8 3 -Sep 85 

3. Psotka 3. Greeno <5c 3. Brown 

(202) 270-5500 (012) 620-4101 

Personnel and Training Research Group of the Office of Naval Research. News & 
Notes, 1982, 2(0), 2-5. 

Procedural-Net Theories of Human Planning and Control 

ONR (667-010) Science Applications, Inc. 3an 78-3un 81 

M.J. Farr M. Atwood $232,086 

(202) 696-0500 (303) 773-6900 

Defense Technical Information Center, work Unit Sum maries. Alexandria, VA: 
DTIC, 1982. T 

The Role of Prijr Knowledge in Operating Devices 

ONR (667-073) University of Arizona 

D. Kieras 
(602) 626-2751 

Office of Naval Research. Psychological , Sciences Division, 1981 Prog rams. 
Arlington, VAs Office of Naval Research, December 1981. ~ 
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Spallation Models for Procedural Learning and Expert Memory 



ONR (157-063) 
MJ. Fair 
(202) 696-0504 



Carnegie- Mel Ion University 
J.R. Anderson 
(012) 578-2781 



Oct 80-Nov 82 
$333,032 



Defense Technical Information Center. Work Unit Summa ries* Alexandria, VA: 
D7IC, 1982. ~ 

Structural Approaches to Procedural Learning from Technical Text and Graphics 



Office of Naval Research. Psychological Sciences Division, 1981 Programs. 
Arlington, VA: Office of Naval Research, December 1981. 

Studies of Skilled Human Performance 



Office of Naval Research. Psychological Sciences Division, 1981 Programs . 
Arlington, VA: Office of Naval Research, December 1981. 



Systems Automation through Artificial Intelligence 

AFOSR 
D. Fox 



Task -Oriented Measurement Technologies 

AFURL (Brooks AFB) 
H.W. Ruck 
(512) 3J6-3608 

Air Force Human Resources Laboratory. AFHRL Annual Report; FY 81 . Brooks 
AFB, TX: AFHRL, 1932. 



ONR (667-052) 



Cornell University 
M.D.. Clock 
(607) 256-1000 



ONR (667-037) 



University of California, 
San Diego 

D.A. Norman & D. Rumelhart 
(710) 052-2230 
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Technical and Engineering Services to Determine Automatic Test Equipment 
Technological Use of AI Applications 



NAVAIRENGCEN 
J. Kunert 
(201) 323-7^63 



Theory of Graphic Representation 

NPRDC 

W.E. Montague 

(7 It) 225-7121 



Navy Personnel Research and Development Center, NPRDC FY-32 Programs . San 
Diego, CAt NPRDC, n.d. 



Tuple and 1 heme Identification in Prose Comprehension 

ONR (667-423) University of Arizona 

D. Kieras 
(602) 626-2751 

Office of Naval Research. Psychological Sciences Division, 1981 Programs . 
Arlington, VA: Office of Naval Research, December 1981. 

Training Aids Developed from Learning Guidelines 

TAEG 
R. Braby 
(305) 6<*6-5673 



Tutoring and Problem-Solving Strategies in Intelligent Computer- Aided 
Instruction 

ONR (154-436) Stanford University 

B. Buchanan <5c W. Clancey 
(<U5) 497-2300 

Office of Naval Research. Psychological Sciences Division, 1981 Programs . 
Arlington, VA: Office of Naval Research, December 1981. 
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Understanding and Executing Instructions 



ONR (134-461) 



Bolt Beranek & Newman, Inc. 
R.L. Smith, Jr. 
(617) 492-48*1 



Office of Naval Research. Psychological Sciences Division. 19? 1 Progr ams. 
Arlington, VA: Office of Naval Research, December 1981. 



Unified Data Base Technology 

AFHRL (Wright-Patterson AFB) 
R.N. Deem 
013) 235-3771 

A c n F ?5 e abu^ 1 , R f«2 urceS Laboratory - AFHRL Annual Report; FY 81. Brooks 
ArB, TXl AFHRL, 1982< 



Voice Technology as the Instructor's Assistant 

NTEC Honeywell Jul 81 -Oct 82 

R. Ahlers 3. Brock $35,000 

(305)646-5130 (612) 378-4403 

S?*!" 5 * Technical formation Center. Work Unii Summaries. Alexandria, VA: 
DTIC) 1982* " 
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